Given data encryption – data mesh in Starship | by Taavi Pungas | Images of Starship Technologies
Gigabytes of data per bag. This is what you get when you send a robotic. That’s a lot – especially if you repeat more than a million times as we have.
But the rabbit hole goes deeper. What is also different is amazing: the robot sensor and image data, user interaction with our apps, what has happened from orders, and much more. And the methods used are varied, from training neural networks to creating visuals for our business partners, and everything in between.
So far, we have been able to address all of these issues with our central data group. In the meantime, the rapid growth has led us to look for new ways to move forward.
We have found that the data mesh paradigm is the best way to move forward. I will explain how the Starship takes the mesh below, but first, let us briefly go over the route and why we decided to go with it.
What is a data mesh?
The data mesh system was originally explained by Zhamak Dehghani. The paradigm is based on this main points: data objects, data domains, data platform, and data management.
The main purpose of the data mesh system has been to assist large corporations to overcome data technology challenges and address issues. It therefore outlines many of the essentials of business, ranging from data type, structure, and security, to leadership and organizational structure. As it stands, only several companies has publicly announced the adherence of the data mesh paradigm – all billions of large businesses. Even so, we think it can be used effectively in small companies, too.
Data mesh in Starship
Do things that work close to the people who are making or using the information
To drive the hyperlocal robotic markets around the world, we need to turn a variety of species into valuable commodities. Data comes from robots (such as telemetry, route options, ETAs), merchants and customers (with their programs, orders, offers, etc.), and all business operations (from short operational operations to global residual resources). ). parts and robots).
Variability in terms of usage is the main reason we have been drawn to the data mesh approach – we want to work with data in close proximity to the people who are creating or using the information. Following the data mesh principles, we hope to meet the diverse needs of our teams and ensure that central drying is relatively easy.
Since Starship has not yet reached commercialization, it is not practical for us to use all parts of the mesh. Instead, we are focused on a simple path that is meaningful to us now and putting us on the right path for the future.
Sales data
Explain what your data assets are – each has its own owners, forms, and users
Applying reasoning to our data is the basis of the whole process. We think of anything that reveals information about other users or methods as a data object. It can display its information in any way: like the BI dashboard, the Kafka header, the interface, the response from the predictive microservice, and much more.
A simple example of data marketing in Starship could be the BI dashboard of leading sites to track the size of their business. A further example would be a self-made way for robotic software developers to send all kinds of information from robots to our data center.
In any case, we do not view our museum (actually the Databricks lakehouse) as a single entity, but as a support center for a number of connected objects. Such small businesses are often owned by scientists / engineers who build and maintain them, not dedicated supervisors.
The owner of the item is expected to know who uses it and what they want to deal with the item – and based on this, describe and live up to the good expectations of the item. As a result, we have begun to look more closely at the links, the components that are necessary for use but difficult to change.
Most importantly, understanding the users and the value that each item makes for them makes it easier to prioritize ideas. This is a must have, for any Affiliate, promoting any program.
Data domains
Integrate your data objects into domains that reflect the company’s structure
Before we knew what kind of data mesh we were, we had been making good use of the form a lightly integrated data scientists for a while in Starship. Similarly, some key groups had a data group member who worked part-time – whatever that meant in a particular group.
We continued to interpret the data areas in accordance with our organizational plan, this time careful to find every aspect of the company. After recording the data items on the domains, we assigned a data group member to select each domain. This person is responsible for managing the entire domain of content in the domain – some belonging to the same person, some by other engineers in the domain team, or some by other members of the data group (such as for access purposes).
There are a number of things we like about setting up our domains. First, now every area of the company has a person who oversees its data design. Considering the mysteries found in each area, this is possible because we have shared the work.
Designing our data objects and connections has also helped us to better understand our data world. For example, when there are more areas than members of a data group (currently 19 vs 7), we are now doing a good job of making sure that each of us is working on related topics. And now we understand that in order to reduce the amount of pain, we need to reduce the amount of connections that are used across the domain.
Finally, a very hidden bonus for using data areas: now we feel we have a way to deal with all kinds of problems. Each time a new route comes along, it becomes clear to everyone where it is needed and who should be running.
There are also some open-ended questions. While some domains lean naturally to reveal much of the origin while others simply use and modify, there are others that have a multiplicity of both. Should these fractions become too large? Or should we have subdomains within adults? We need to make these decisions along the way.
Data page
Encourage people who are making your data products stable without interference
The purpose of the data platform in Starship is straightforward: it is possible for a single data person (usually a data scientist) to manage end-of-life data, for example, to maintain a central data group on a day-to-day operations. This is required to provide domain design experts and data scientists with the best tools and building blocks for their products.
Does that mean you need a complete set of data on the mesh path? Not really. Our data platform team has one engineer who spends half of their time in the domain. The main reason we can rely heavily on data platform technology is the choice of Spark + Databricks as the basis for our data platform. Our past architecture, traditional archeology has placed engineering that is important to us because of the diversity of our regions.
We have found that it is useful to clearly distinguish the pile of data between the components that are part of the platform as opposed to anything else. Some examples of what we offer to domain groups as part of our data platform:
- Databricks + Spark as a workspace and portable integrated platform;
- single liner input functions, for example from Mongo collections or Kafka themes;
- Airflow model for preparing data pipelines;
- templates for building and sending predictive models like microservices;
- tracking the cost of data items;
- BI & visual aids.
As a general rule, our goal is to design how it can make sense in our modern world – even the little things that we know will not stay the same forever. As long as it supports the current harvest, and does not include any part of the process, we are happy. And of course, some things are missing from the platform right now. For example, tools that verify data type, data availability, and data line are things we have left for the future.
Data control
Its powerful owner aided by feedback
Having a small group of people is a very important aspect of governance, for example it is easy to make decisions. On the other hand, our most important question of governance is directly related to our size. If there is one person for data on any enemy, he cannot expect to be an expert in all that is possible. However, only people who have enough information about their location. How do we maximize the opportunity for them to make good decisions in their place?
Our answer: through personal culture, discussions, and comments within the group. We have generously borrowed from the management philosophy on Netflix and developed the following:
- personal responsibility for results (material and regional);
- seeking various ideas before making decisions, especially those that affect other areas;
- asking for answers and reviews of code as the best way and opportunity for your growth.
We have also made a number of agreements regarding how we behave, how to write down what we do well (including naming), and so on.
These principles also apply outside of the “building” function of our data group – which has been the purpose of this post. Obviously, there are many than to provide data objects the way our scientists make a profit on a company.
One last thought on leadership – we will continue to review our working strategies. There will be no “better” way to do things and we know we need to change over time.
Concluding remarks
This is it! These were 4 starting mesh ideas as they are used in Starship. As you can see, we have found a way to access data mesh that fits us as a long-term growth company. If it sounds interesting in your article, I hope reading about our experience has helped you.
If you want to join our work, see our job page on the list of open positions. Or check our Youtube channel to learn more about our world-class robots project.
Contact me if you have any questions or suggestions and we will learn from each other!
Source link
