10k

设计模式之美-课程笔记53-项目实战3-设计实现一个支持自定义规则的灰度发布组件

Published on

需求场景

  1. 之前说到的我们的公共组件平台相对于业务时一个独立的服务；
  2. 发现基于RPC的服务不好用，想替换为RESTful的服务；-》 需要修改公共服务平台的代码和调用方代码
  3. 保险起见需要灰度替换老的服务。
  4. 需要先让流量小不重要的业务先试用，一段时间验证没有啥问题后再逐步推全。
    1. 问题：出问题回滚会导致调用方部分业务功能也一起回滚。
    2. 问题：不回滚，直接替换RESTfull的代码，然后基于这个代码再做修改，merge back之后可能会引起冲突（RESTful框架开放过程中进入的功能代码在开发分支没有）
  5. 解决回滚成本高的问题：增加功能开关，新老逻辑并存。（为了开关导致的部署上线，可以将这个flag放在配置文件）
  6. 如何实现灰度？
    1. 决定灰度的对象： 时间戳。业务ID，按照区间、比例或者具体值来做灰度。
    2. 比如要根据用户ID做灰度：先配置用户ID123， 456， 789的查询调用新接口，没问题后再让1000-1100之间的查询新接口；如果还是没问题我们再继续扩大范围让10%用户的查询请求调用新接口。
  7. 对于复杂功能，核心功能即使不做灰度也建议通过功能开关灵活控制上下线。再不需要部署和重启系统的情况，做到快速回滚或者切换。

需求分析

  1. 这个需求的灰度是代码级别的灰度，平时所说的一般是系统层面（不同版本的灰度让新版本平滑上线）或者产品层面（ABTesting 给不同的功能对比用户体验）的灰度。

功能性需求

  1. 开关
  2. ~~灰度对象~~ 灰度功能标识ID
  3. rule：对象ID（值、range、比例）
  4. 灰度配置加载，灰度判断逻辑

非功能性需求

  1. 易用性：非侵入或容忍一定程度的侵入（因为是代码层的灰度，控制代码，有if-Else），简单配置，热更新
  2. 容错：对于异常的处理
  3. 可用：不影响原来业务的正常使用，灰度失效时候traffic全部走老逻辑。
  4. 性能：不影响业务本身-》灰度对象是否在灰度区间的判定尽可能高效
  5. 扩展性、灵活性：支持不同的格式，不同的存储方式的配置文件。-》 如何支持更复杂的灰度配置？（20天内退货次数少于10次的用户）

框架设计思路

  1. 复杂灰度规则
    1. 规则引擎-》 配置文件可以调用Java代码；-》 引入第三方框架增加复杂度。
    2. 编程规则编程实现（灵活，但更新规则需更新代码）
  2. 灰度规则热更新
    1. 定时器：定时读取配置并加载最新的配置到内存；
    2. 读取和加载并发执行：热加载不能影响灰度服务

功能需求整理

1. 灰度规则的格式和存储方式

同限流的格式、存储定义方式

2. 灰度规则的语法格式

具体值、区间值、比例值、复杂规则

3. 灰度规则的内存组织方式

灰度规则的数据结构支持快速判定是否在灰度对象中

4. 灰度规则的热更新

基本功能实现

// 代码目录结构
com.xzg.darklaunch
  --DarkLaunch(框架的最顶层入口类)
  --DarkFeature(每个feature的灰度规则)
  --DarkRule(灰度规则)
  --DarkRuleConfig(用来映射配置到内存中)

// Demo示例
public class DarkDemo {
  public static void main(String[] args) {
    DarkLaunch darkLaunch = new DarkLaunch();
    DarkFeature darkFeature = darkLaunch.getDarkFeature("call_newapi_getUserById");
    System.out.println(darkFeature.enabled());
    System.out.println(darkFeature.dark(893));
  }
}

// 灰度规则配置(dark-rule.yaml)放置在classpath路径下
<!--对应DarkRuleConfig-->
features:                       
- key: call_newapi_getUserById  <!--对应DarkFeatureConfig-->
  enabled: true
  rule: {893,342,1020-1120,%30}
- key: call_newapi_registerUser <!--对应DarkFeatureConfig-->
  enabled: true
  rule: {1391198723, %10}
- key: newalgo_loan             <!--对应DarkFeatureConfig-->
  enabled: true
  rule: {0-1000}

DarkLaunch

  1. 组装、串联操作流程
  2. 读取配置文件（DarkRuleConfig），构建支持快速查询的数据结构（DarkRule）；
  3. 还负责定期热更新灰度规则。
public class DarkLaunch {
  private static final Logger log = LoggerFactory.getLogger(DarkLaunch.class);
  private static final int DEFAULT_RULE_UPDATE_TIME_INTERVAL = 60; // in seconds
  private DarkRule rule;
  private ScheduledExecutorService executor;

  public DarkLaunch(int ruleUpdateTimeInterval) {
    loadRule();
    this.executor = Executors.newSingleThreadScheduledExecutor();
    this.executor.scheduleAtFixedRate(new Runnable() {
      @Override
      public void run() {
        loadRule();
      }
    }, ruleUpdateTimeInterval, ruleUpdateTimeInterval, TimeUnit.SECONDS);
  }

  public DarkLaunch() {
    this(DEFAULT_RULE_UPDATE_TIME_INTERVAL);
  }

  private void loadRule() {
    // 将灰度规则配置文件dark-rule.yaml中的内容读取DarkRuleConfig中
    InputStream in = null;
    DarkRuleConfig ruleConfig = null;
    try {
      in = this.getClass().getResourceAsStream("/dark-rule.yaml");
      if (in != null) {
        Yaml yaml = new Yaml();
        ruleConfig = yaml.loadAs(in, DarkRuleConfig.class);
      }
    } finally {
      if (in != null) {
        try {
          in.close();
        } catch (IOException e) {
          log.error("close file error:{}", e);
        }
      }
    }

    if (ruleConfig == null) {
      throw new RuntimeException("Can not load dark rule.");
    }
    // 更新规则并非直接在this.rule上进行，
    // 而是通过创建一个新的DarkRule，然后赋值给this.rule，
    // 来避免更新规则和规则查询的并发冲突问题
    DarkRule newRule = new DarkRule(ruleConfig);
    this.rule = newRule;
  }

  public DarkFeature getDarkFeature(String featureKey) {
    DarkFeature darkFeature = this.rule.getDarkFeature(featureKey);
    return darkFeature;
  }
}

为了避免查询和更新冲突，先创建一个新的rule对象，创建完成过后在替换老对象。

DarkRuleConfig(包含DarkFeatureConfig内部类)

  1. 将灰度规则映射到内存中
public class DarkRuleConfig {
  private List<DarkFeatureConfig> features;

  public List<DarkFeatureConfig> getFeatures() {
    return this.features;
  }

  public void setFeatures(List<DarkFeatureConfig> features) {
    this.features = features;
  }

  public static class DarkFeatureConfig {
    private String key;
    private boolean enabled;
    private String rule;
    // 省略getter、setter方法
  }
}

DarkRule

public class DarkRule {
  private Map<String, DarkFeature> darkFeatures = new HashMap<>();

  public DarkRule(DarkRuleConfig darkRuleConfig) {
    List<DarkRuleConfig.DarkFeatureConfig> darkFeatureConfigs = darkRuleConfig.getFeatures();
    for (DarkRuleConfig.DarkFeatureConfig darkFeatureConfig : darkFeatureConfigs) {
      darkFeatures.put(darkFeatureConfig.getKey(), new DarkFeature(darkFeatureConfig));
    }
  }

  public DarkFeature getDarkFeature(String featureKey) {
    return darkFeatures.get(featureKey);
  }
}

DarkFeature

public class DarkFeature {
  private String key;
  private boolean enabled;
  private int percentage;
  private RangeSet<Long> rangeSet = TreeRangeSet.create();

  public DarkFeature(DarkRuleConfig.DarkFeatureConfig darkFeatureConfig) {
    this.key = darkFeatureConfig.getKey();
    this.enabled = darkFeatureConfig.getEnabled();
    String darkRule = darkFeatureConfig.getRule().trim();
    parseDarkRule(darkRule);
  }

  @VisibleForTesting
  protected void parseDarkRule(String darkRule) {
    if (!darkRule.startsWith("{") || !darkRule.endsWith("}")) {
      throw new RuntimeException("Failed to parse dark rule: " + darkRule);
    }

    String[] rules = darkRule.substring(1, darkRule.length() - 1).split(",");
    this.rangeSet.clear();
    this.percentage = 0;
    for (String rule : rules) {
      rule = rule.trim();
      if (StringUtils.isEmpty(rule)) {
        continue;
      }

      if (rule.startsWith("%")) {
        int newPercentage = Integer.parseInt(rule.substring(1));
        if (newPercentage > this.percentage) {
          this.percentage = newPercentage;
        }
      } else if (rule.contains("-")) {
        String[] parts = rule.split("-");
        if (parts.length != 2) {
          throw new RuntimeException("Failed to parse dark rule: " + darkRule);
        }
        long start = Long.parseLong(parts[0]);
        long end = Long.parseLong(parts[1]);
        if (start > end) {
          throw new RuntimeException("Failed to parse dark rule: " + darkRule);
        }
        this.rangeSet.add(Range.closed(start, end));
      } else {
        long val = Long.parseLong(rule);
        this.rangeSet.add(Range.closed(val, val));
      }
    }
  }

  public boolean enabled() {
    return this.enabled;
  }

  public boolean dark(long darkTarget) {
    boolean selected = this.rangeSet.contains(darkTarget);
    if (selected) {
      return true;
    }

    long reminder = darkTarget % 100;
    if (reminder >= 0 && reminder < this.percentage) {
      return true;
    }

    return false;
  }

  public boolean dark(String darkTarget) {
    long target = Long.parseLong(darkTarget);
    return dark(target);
  }
}

上面的代码可以用工厂+策略模式来优化多个if-else的逻辑，来更好的满足OCP。

RuleConfig和FeatureConfig都被抽离出来，并没有直接在Rule和Feature上面去做解析。

添加、优化功能

  1. 增加功能支持复杂规则配置
// 第一步的代码目录结构
com.xzg.darklaunch
  --DarkLaunch(框架的最顶层入口类)
  --DarkFeature(每个feature的灰度规则)
  --DarkRule(灰度规则)
  --DarkRuleConfig(用来映射配置到内存中)

// 第二步的代码目录结构
com.xzg.darklaunch
  --DarkLaunch(框架的最顶层入口类，代码有改动)
  --IDarkFeature(抽象接口)
  --DarkFeature(实现IDarkFeature接口，基于配置文件的灰度规则，代码不变)
  --DarkRule(灰度规则，代码有改动)
  --DarkRuleConfig(用来映射配置到内存中，代码不变)

IDarkFeature

public interface IDarkFeature {
  boolean enabled();
  boolean dark(long darkTarget);
  boolean dark(String darkTarget);
}

DarkRule

为了避免配置文件的灰度规则热更新时候覆盖掉编程实现的灰度规则，对从配置文件中加载的灰度规则和编程实现的灰度规则分开存储。

public class DarkRule {
  // 从配置文件中加载的灰度规则
  private Map<String, IDarkFeature> darkFeatures = new HashMap<>();
  // 编程实现的灰度规则
  private ConcurrentHashMap<String, IDarkFeature> programmedDarkFeatures = new ConcurrentHashMap<>();

  public void addProgrammedDarkFeature(String featureKey, IDarkFeature darkFeature) {
    programmedDarkFeatures.put(featureKey, darkFeature);
  }

  public void setDarkFeatures(Map<String, IDarkFeature> newDarkFeatures) {
    this.darkFeatures = newDarkFeatures;
  }

  public IDarkFeature getDarkFeature(String featureKey) {
    IDarkFeature darkFeature = programmedDarkFeatures.get(featureKey);
    if (darkFeature != null) {
      return darkFeature;
    }
    return darkFeatures.get(featureKey);
  }
}

DarkLaunch

public class DarkLaunch {
  private static final Logger log = LoggerFactory.getLogger(DarkLaunch.class);
  private static final int DEFAULT_RULE_UPDATE_TIME_INTERVAL = 60; // in seconds
  private DarkRule rule = new DarkRule();
  private ScheduledExecutorService executor;

  public DarkLaunch(int ruleUpdateTimeInterval) {
    loadRule();
    this.executor = Executors.newSingleThreadScheduledExecutor();
    this.executor.scheduleAtFixedRate(new Runnable() {
      @Override
      public void run() {
        loadRule();
      }
    }, ruleUpdateTimeInterval, ruleUpdateTimeInterval, TimeUnit.SECONDS);
  }

  public DarkLaunch() {
    this(DEFAULT_RULE_UPDATE_TIME_INTERVAL);
  }

  private void loadRule() {
    InputStream in = null;
    DarkRuleConfig ruleConfig = null;
    try {
      in = this.getClass().getResourceAsStream("/dark-rule.yaml");
      if (in != null) {
        Yaml yaml = new Yaml();
        ruleConfig = yaml.loadAs(in, DarkRuleConfig.class);
      }
    } finally {
      if (in != null) {
        try {
          in.close();
        } catch (IOException e) {
          log.error("close file error:{}", e);
        }
      }
    }

    if (ruleConfig == null) {
      throw new RuntimeException("Can not load dark rule.");
    }
    
    // 修改：单独更新从配置文件中得到的灰度规则，不覆盖编程实现的灰度规则
    Map<String, IDarkFeature> darkFeatures = new HashMap<>();
    List<DarkRuleConfig.DarkFeatureConfig> darkFeatureConfigs = ruleConfig.getFeatures();
    for (DarkRuleConfig.DarkFeatureConfig darkFeatureConfig : darkFeatureConfigs) {
      darkFeatures.put(darkFeatureConfig.getKey(), new DarkFeature(darkFeatureConfig));
    }
    this.rule.setDarkFeatures(darkFeatures);
  }

  // 新增：添加编程实现的灰度规则的接口
  public void addProgrammedDarkFeature(String featureKey, IDarkFeature darkFeature) {
    this.rule.addProgrammedDarkFeature(featureKey, darkFeature);
  }

  public IDarkFeature getDarkFeature(String featureKey) {
    IDarkFeature darkFeature = this.rule.getDarkFeature(featureKey);
    return darkFeature;
  }
}

Demo

灰度规则配置(dark-rule.yaml)，放到classpath路径下

features:
- key: call_newapi_getUserById
  enabled: true
  rule: {893,342,1020-1120,%30}
- key: call_newapi_registerUser
  enabled: true
  rule: {1391198723, %10}
- key: newalgo_loan
  enabled: true
  rule: {0-100}

编程实现的灰度规则

public class UserPromotionDarkRule implements IDarkFeature {
  @Override
  public boolean enabled() {
    return true;
  }

  @Override
  public boolean dark(long darkTarget) {
    // 灰度规则自己想怎么写就怎么写
    return false;
  }

  @Override
  public boolean dark(String darkTarget) {
    // 灰度规则自己想怎么写就怎么写
    return false;
  }
}

Demo

public class Demo {
  public static void main(String[] args) {
    DarkLaunch darkLaunch = new DarkLaunch(); // 默认加载classpath下dark-rule.yaml文件中的灰度规则
    darkLaunch.addProgrammedDarkFeature("user_promotion", new UserPromotionDarkRule()); // 添加编程实现的灰度规则
    IDarkFeature darkFeature = darkLaunch.getDarkFeature("user_promotion");
    System.out.println(darkFeature.enabled());
    System.out.println(darkFeature.dark(893));
  }
}
Thoughts? Leave a comment