And they’re leaving it in my hands 🙃

Anyone have any suggestions for tools/technologies to look into? It will most likely take the form of csv extracts from system A with fields/objects to be mapped to system B.

Also, this question is truly an instance of how the internet is a fucking wasteland. I could’ve asked this question 5 or 6 years ago on Reddit and gotten something meaningful, but today I would get replies from tons of company accounts trying to sell me their garbage solutions based on LLM wrappers

  • JoeByeThen [he/him, they/them]@hexbear.net
    link
    fedilink
    English
    arrow-up
    13
    ·
    11 days ago

    Without know specifics kinda hard to know what to recommend but you’re probably going to need a basic structure of properly backing up your dumps from system A, build a python script to convert the schema of system A to the schema of system B and properly store a backup of your new files to be uploaded to system B. Depends on how complicated you wanna make the process.

    If you’re starting from scratch, you’re basically the trailblazer here. Is your company giving you a budget or a team to build the workflow?

  • leftAF [comrade/them]@hexbear.net
    link
    fedilink
    English
    arrow-up
    10
    ·
    edit-2
    11 days ago

    Use dbt, host it on whatever they got (vSphere, cloud VMs, w/e) or schedule it the same way their “Devops” or “CICD” pipelines are orchestrated. Makes it easier if you wind up having a bunch of workers move data in parallel during the migration window or if you need to do a rolling refresh of the destination for validation or something.

    If you can get access to the source database, query the INFORMATION_SCHEMA views (select * from information_schema.columns) to get the data types of the fields in the source system. Same for the destination it sounds like, but once you have the columns and tables present in both systems you can probably start mapping it easily or chatbots probably do OK at it.

    • Hohsia [any]@hexbear.netOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 days ago

      What is dbt? Dealing with azure here too, but no access to anything other than SOQL (platform sits on top of salesforce)

  • BigWeed [none/use name]@hexbear.net
    link
    fedilink
    English
    arrow-up
    7
    ·
    11 days ago

    I would start by documenting the schemas (data types and optional fields). Then use shell scripts to transform or join any data to be ingested to another system. Not sure what you’re working with so it’s hard to give any more concrete advice.

    • Hohsia [any]@hexbear.netOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 days ago

      It’s a platform that sits on top of salesforce

      Bleh hate it lol would be so sick if I could query SQL information schema tables