Node 16 update & test

CI/CD for dev and test environment, node affinity

Fix linting errors if present, fix linting warnings

Review state of testing

Review state of open issues

Review state of open PRs

Fix security audit reports

• npm audit --production - list vulnerable packages while ignoring dev dependencies

• npm ls <package-name> - trace vulnerable packages to the base level packages that are using them

Typescript - review appropriateness of typescript for this service if the dev knows typescript

Microservices implementation structural review

Schema support for db-migrate

• We decided to assign search_path to each microservice user (service and migration users) in terraform to resolve this need.

Developer Productivity

• Codeowners file and agreements

• Document observability tools and access

  • ELK: kubernetes.container.image :"greenstand/treetracker-api:1.4.0"

    • deployed image version can be found by visiting the root path of any deployed api

    • (such as )
• Document about how deployment to dev environment works.

• Figure out if dicpher can work with node16 or if there is a replacement (npm run decrypt)

  • Password for dipher is pinned

• Sealed secrets: treetracker-infrastructure/sealed-secrets/scripts/create-database-secret.sh

• resolve all lint warnings

  • Update lint settings to our likely - are there rules we want to enable?

• remove co-mocha?

1. Release treetracker API into all envs

2. Web map query schema & consumer into all env

   1. Release

   2. Validate data flow for raw captures

   3. Validate data flow for captures

   4. Validate or implement cluster generation

   5. Validate or implement

3. Implement web map for raw captures in field data schema

4. Finalize and Release Capture Matching Tool

5. Finalize and Release Earnings Tool

6. Finalize and Release Stakeholder Tool

7. Migrate admin panel users to keycloak

8. Migrate web map to captures table in treetracker schema

Migration Sequence

1. Migrate all legacy 'tree' table data into the fielddata.raw_capture table

   1. Includes creation session and device records for these captures

2. Admin panel application: split capture verification tool from capture search tools.

3. Go through the migration process for capture verification

   1. Documentation:

   2. Steps

4. Migrate all remaining approved legacy 'tree' table data into the treetracker.capture table

   1. Apply to dev

   2. Apply to test

5. Consolidation step to fix all planting organization assignments

   1. Priority between planting org per planter, per tree, and reported in field data to be determined

   2. Clean up all capture records to have valid organization assignments in the stakeholder tool.

6. Update admin panel to using treetracker API for capture search tools.

7. *Now, ready to get the capture matching tool working in deployment*

8. GIS task: Identify very small geographic area in Freetown as a pilot for capture matching, and supply this as a spatial shape file (.geojson or .shp)

9. GIS task: Identify all the capture records in the small geographic area that are the initial captures of the tree.

10. Extract production data from small geographic area and add that data to dev and test database.

11. Data task: insert treetracker.tree records for each initial capture. Now we have trees to capture match against.

    1. Perform in dev, test, and prod

12. Implement capture matching frontend feature to allow filtering trees and captures by named geographic area. We will filter for initial use by the small geographic area defined above, which should be loaded into the regions API.

13. Now we can deploy the capture matching tool to the beta infrastructure and try it out.

1. Some legacy planter registrations reported to not have device identifier.

   1. This is inconvenient but not fixable with the v2 bulk pack format

‘Ground user’ and ‘individual’ are synonyms. Also ‘person’

A mobile app user registers a wallet when using the mobile phone. Wallet registrations are uploaded along with their corresponding captures. A wallet is created for the wallet registration if it does not already exist. A grower account is created for each wallet, to track the identity of the person until they can be verified. Once a person is verified, a person record is created for them in the stakeholder system. Person records can receive payments for treetracker services.

Diagram for contract tables:

Capture Matching Release Project

Capture Match road map 2021:

Webmap query service

Domain Services

Treetracker API

Source Code: https://github.com/Greenstand/treetracker-api

Mounted Endpoint: https://{env}-k8s.treetracker.org/treetracker/

Internal Endpoint: http://treetracker-api.treetracker-api

Field Data API

Source Code: https://github.com/Greenstand/treetracker-field-data

Mounted Endpoint: https://{env}-k8s.treetracker.org/field-data/

Internal Endpoint: http://treetracker-field-data.field-data-api/

Source Code:

Mounted Endpoint:

Internal Endpoint:

Source Code:

Mounted Endpoint:

Internal Endpoint:

Source Code:

Mounted Endpoint:

Internal Endpoint:

Source Code:

Mounted Endpoint:

Internal Endpoint:

Source Code:

CDN Endpoint: , also tiles2, tiles3, tiles4

Mounted Endpoint:

Source Code:

Mounted Endpoint:

Internal Endpoint:

Source Code:

This queue message consumer exposes no API

Source Code:

This queue message consumer exposes no API

Source Code:

This scheduled job exposes no API

Source Code:

Mounted Endpoint: not mounted

Internal Endpoint:

Source Code:

Mounted Endpoint: not mounted

Internal Endpoint:

Source Code:

Mounted Endpoint:

Internal Endpoint:

Source Code:

Mounted Endpoint:

Internal Endpoint:

1. Update to 'verified' status requested to treetracker API

2. Start transaction against treetracker schema to insert a new capture record

   1. If the record already exists, that's OK, just accept.

3. Call legacy API and update capture (trees table record) to approved

4. Commit transaction on treetracker schema

5. Call field data API and update raw_capture record to approved

if (3) fails, we rollback the treetracker schema transaction, therefore the raw_capture record remains unprocessed, and will be reprocessed by an operator in the future, should bring data into consistency.

if (4) fails, same as if (3) fails, we no update. Will be reprocessed

if (5) fails, the record will be reprocessed

When we POST to captures/, send the UUID of the raw_capture to become the UUID of the approved capture record - these should always be unique.

• Allows for easy reprocessing when there are outages

• Allows for multiple concurrent POST messages to be collapsed and only create 1 capture record

• Only access the API, do not perform database check

  • First write seeded tests for GET endpoints

  • Then write tests for create and update that use GET endpoint to validate functionality

1. We will control available authors manually at first, by populating the authors table

2. We will control access to admins using our normal process

3. For announce and survey messages, we look up associated Grower Accounts in the treetracker API, and then match those against enabled authors

   1. We need to extend this approach to respect the organizational hierarchy, which may mean a query service for grower accounts.

   2. A query service for grower accounts would also more easily respect the region filters

Handler

• HTTP Domain

• All error and success codes

• API payload validation

• Calls a service or a model function

• Orchestration between services and the domain model

  • Database session

  • External APIs

• Domain logic

• Accesses repositories

• Allowances

• Accesses the database, performs CRUD operations

• One repository for each database table in RDMS

Goal: Admin panel to not read trees table directly. Build new treetracker service with API to create capture records.

Build a new service to be the source of truth of all approved captures.

This service is to own all the capture records along with tagging and other details created during the approval in the admin panel.

• Service is already created, known as treetracker-api.

• Some verbs (GET/POST/PATCH) may be missing but capture, tree, and planter resources have been defined.

• We may need to add additional search parameters as needed by other services or for domain goals.

• This appears to be implemented, but has not been tested or operated

• We added planter (ground_user) and tree to this service already. That will be the scope of the service. The service is called treetracker serivce, and is stored in the treetracker-api repository

• Data populated in the admin panel verify tool, should ONLY be raw captures from the field data service that has not been verified (status=unprocessed)

• This will require the admin panel frontend team to split the current verify tool into a 'verification' tool and a 'capture edit' tool.

• Only verified captures can be 'edited' through the 'capture edit' tool, and only very limited fields can be edited. mostly this is tagging.

• (1) POST to treetracker/capture

• (2) PATCH to the legacy admin panel API

• (3) PATCH back to the field service to mark as approved/rejected in field data schema.

• Remove the feature that emitted event for approved captures from Change 1 earlier (in the legacy API). The reason we will write to trees table from admin panel (or orchestration layer) is for back-ward compatability with web-map.

• This appears to already be complete. Need to be tested and operated.

Migrate data from trees table in main db to captures table in the new captures service of all approved tree entries.

• ETL job. Can be implemented as one-off airflow job.

Update token generation process and refer to ids from captures instead of trees as reference association.

• This refers to the token minting scripts, now being run in airflow

• capture_id for each token needs to reference the id of treetracker.capture rather than public.trees. This also requires updating the HATEOAS link in the token payload (wallet API) to point to the new capture service.

• Update all existing tokens to use treetracker.capture.id as capture_id

• The events consumed in web-map layer will trigger api calls to get the tokens impacted and update the capture view in the web-layer to assign the updated wallet owner.

• Requires endpoint to look up all tokens in any transfer by transfer id.

• Discuss how organizational hierarchy is captured in the webmap schema (JSON blob or table structure)

• Breaking changes will trigger a new major revision, and a new deployment mount point

  • Frontend engineers will be responsible for migrating to the new mount point

  • If database changes are necessary, if the service is not in production it's ok to make a breaking change. Otherwise check with engineering leadership.

Field data schem

Ingestion Process by Entity Type

wallet_registration

Wallet registrations need to be stored in the field data schema in the wallet_registrations table, but they also need to be used to upsert (create or update) grower account records in the treetracker API. A wallet registration also needs the grower account id field populated.

Suggested workflow:

1. Bulk pack transformer calls PUT on grower_account in treetracker API for grower account with identifier that matches wallet

   1. If account exists, update

      1. Only update the name, phone, and email for now

   2. Else, insert

      1. Insert identifier, name, phone, email, and photo

2. POST to /planter path on bulk-pack-transformer-1 to populate legacy schema

3. Populate grower account id in wallet registration data, and POST into field data api

   1. If wallet registration id is already present, field data api will do nothing (return 200)

4. Return 200

device configuration records are simply inserted in v2, but in v1 they were upserted.

1. POST device configuration payload to bulk-pack-transformer-1

   1. bulk-pack-transformer-1 returns 200 if already present

2. POST device configuration payload to field data API

session does not exist in the v1 bulk-pack-transformer.

1. POST to session path on field data API

   1. field data API returns 200 if already present

2. Return 200

1. POST capture data to bulk-pack-transformer-1

   1. If data exists this API returns 200, else a new record is inserted

   2. This API also propagates the record to field data API, where a queue message is emitted

Bulk Pack Format v2 :

Session

• track_url

• organization

Wallet Registration

• phone or email is nullable, but not both

Raw Capture

• session_id is nullable for v1 endpoints only

• reference_id

• note

Device Configuration

• No nullable fields

Insert 'tree' as 'capture' will fail until corresponding device and registration records have been inserted, because there's no session Once device and registration records are inserted, then it is possible to insert a session id using SESSION UUID = device_identifier + planter_identifier In v2 bulk pack endpoint /v1/tree is responsible for ensuring that the session record exists for a v1 tree. This session can then be used as the session for inserting the v1 'tree' record as a v2 'capture' record

1. process sends tree record to /v1/tree in v2

2. v2 calculates SESSION UUID = device_identifier + planter_identifier using uuid-string

3. v2 checks for a session that matches this

Old versions:

• greenstand/bulk-pack-processor:1.2.7

• greenstand/bulk-pack-transformer:1.6.2

treetracker.grower_account.wallet

• is the same as messaging.author.handle

• is the same as (public.planter.phone || public.planter.email)

messaging.author.handle

• does not necessarily have a corresponding treetracker.grower_account.wallet

• does not have a matching (public.planter.phone || public.planter.email) if it does not have a treetracker.grower_account.wallet

• Disable CORS for development by importing "cors" and then use it to disable cors for 'development' in each microservice

  if (process.env.NODE_ENV === 'development') { log.info('disable cors'); app.use(cors()); }

  EXAMPLE w/ cors:

  EXAMPLE w/ header options:

**Capture**

**Tree**

**Tag**

**Capture_Tag**

**Tree_Tag**

**Grower_Account**

**Status Enumeration** active deleted Session - Field Data Schema Device Configuration - Field Data Schema