Update script that creates managed rule list used by Config

[kbalk]

The script, scripts/pull_down_aws_managed_rules.py, used to download markdown files from an AWS repo. Those markdown files contained info on managed rules, similar to what appears on AWS official documentation, but in a format that's easier to parse. However, the repo containing markdown files was archived on June 15, 2023 as AWS doesn't want to maintain a separate list of the managed rules (read their explanation here).

So scripts/pull_down_aws_managed_rules.py has been updated to pull the managed rule info from AWS documentation websites and the parsed results (moto/config/resources/aws_managed_rules.json) has been updated to match that documentation. The updates in the last year for manged rules appear to include adding another label, "Resource Types", changes to regions and a few new services.

[kbalk]

Just for the record, I noticed the following differences between the old list of markdown documents and the resource list in the AWS documentation (https://docs.aws.amazon.com/config/latest/developerguide/managed-rules-by-aws-config.html):

21a22
> ATHENA_WORKGROUP_ENCRYPTED_AT_REST
48a50
> CLOUDFRONT_S3_ORIGIN_ACCESS_CONTROL_ENABLED
62c64
< CLOUD_TRAIL_ENABLED
---
> CLOUDTRAIL_ENABLED
76a79
> CUSTOM_SCHEMA_REGISTRY_POLICY_ATTACHED
82a86,87
> DOCDB_CLUSTER_BACKUP_RETENTION_CHECK
> DOCDB_CLUSTER_ENCRYPTED
95a101
> EC2_CLIENT_VPN_NOT_AUTHORIZE_ALL
99c105
< EC2_INSTANCE_MANAGED_BY_SSM
---
> EC2_INSTANCE_MANAGED_BY_SYSTEMS_MANAGER
153a160
> ELASTICACHE_SUPPORTED_ENGINE_VERSION
194,195c201,202
< INCOMING_SSH_DISABLED
< INSTANCES_IN_VPC
---
> RESTRICTED_SSH
> EC2_INSTANCES_IN_VPC
204a212
> MACIE_STATUS_CHECK
205a214,215
> MQ_AUTOMATIC_MINOR_VERSION_UPGRADE_ENABLED
> MQ_CLOUDWATCH_AUDIT_LOGGING_ENABLED
207c217
< MULTI_REGION_CLOUD_TRAIL_ENABLED
---
> MULTI_REGION_CLOUDTRAIL_ENABLED
208a219,227
> NEPTUNE_CLUSTER_BACKUP_RETENTION_CHECK
> NEPTUNE_CLUSTER_CLOUDWATCH_LOG_EXPORT_ENABLED
> NEPTUNE_CLUSTER_COPY_TAGS_TO_SNAPSHOT_ENABLED
> NEPTUNE_CLUSTER_DELETION_PROTECTION_ENABLED
> NEPTUNE_CLUSTER_ENCRYPTED
> NEPTUNE_CLUSTER_IAM_DATABASE_AUTHENTICATION
> NEPTUNE_CLUSTER_SNAPSHOT_ENCRYPTED
> NEPTUNE_CLUSTER_SNAPSHOT_PUBLIC_PROHIBITED
> NETFW_LOGGING_ENABLED
226a246
> RDS_CLUSTER_ENCRYPTED_AT_REST
254c274
< RESTRICTED_INCOMING_TRAFFIC
---
> RESTRICTED_COMMON_PORTS
296a317
> STEP_FUNCTIONS_STATE_MACHINE_LOGGING_ENABLED

[codecov]

Codecov Report

Merging #6823 (584b6e7) into master (ab8bf21) will increase coverage by 0.00%.
Report is 8 commits behind head on master.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6823   +/-   ##
=======================================
  Coverage   96.20%   96.20%           
=======================================
  Files         816      816           
  Lines       80042    80062   +20     
=======================================
+ Hits        77004    77024   +20     
  Misses       3038     3038           

Flag	Coverage Δ
servertests	37.01% <ø> (-0.01%)	⬇️
unittests	96.14% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 5 files with indirect coverage changes

[kbalk]

@bblommers This is delicate code in the sense that it can break if AWS changes the HTML for the documentation pages. I used the lxml package for HTML parsing since that package is already pulled in with the test requirements. I could add lxml to the dev requirements if you wish.

The HTML is more consistent than the markdown files. Previously I had to handle odd cases in the markdown text, but with the HTML there were no special checks needed.

[bblommers]

Looks good to me - thank you for updating this @kbalk!

I'll see if I can make a weekly cron job out of this, so the rules are always up-to-date (+ we get a more timely reminder if it does break..)

[kbalk]

I'll see if I can make a weekly cron job out of this, so the rules are always up-to-date (+ we get a more timely reminder if it does break..)

Monthly or quarterly might be good enough. The list of managed rules was last updated a year ago and no one reported issues with it.

I was looking at the config code and it doesn't do a lot with the managed rules at the moment because boto3 didn't do much either (and maybe still doesn't). If that should change, than keeping the list of managed rules up-to-date would be more important.

Also, "Resource Types" was added to the managed rules info, but I don't know how that's being used by AWS.