Posted in

How does a Transformer handle non – textual data?

If you’ve spent any time in AI lately, you’ve definitely heard of Transformers—right now, they’re the backbone of everything from ChatGPT to image generators, and everyone assumes they’re only for text, right? Wait, no. Let’s be real: Transformers don’t care if your data is words, numbers, pixels, or even sensor readings. As a Transformer supplier, I get asked this question all the time: “How does a Transformer actually handle non-textual data without breaking?” Spoiler: it’s not magic—it’s a few clever workarounds, and most of the time, people don’t even realize we’re using these every day. Transformer

Let’s start with the basics: what even is a Transformer, at its core? Yeah, it’s that model architecture with attention mechanisms, right? Originally built for machine translation back in 2017, but the big thing is it doesn’t process data as full blocks—wait no, actually, it splits sequences into little chunks called tokens. For text, tokens are just words, subwords, or even individual characters if you’re being nitpicky. But for non-text data, there’s no “words” to grab, so we have to turn whatever you’ve got into something the model understands. That’s where embeddings come in, and that’s where we as a Transformer supplier add most of the value for non-text use cases.

Take images, for example. If you feed a photo straight into a Transformer, it’s just a grid of pixels—each pixel is a number for red, green, blue, that’s it. So we split the image into small, square patches, like 16×16 pixels each. Each patch gets flattened into a single vector, and that’s our token. No words here, just a bunch of pixel patches turned into numbers the model can work with. Wait, but hold on—without attention, that’s just a big bag of patches. The attention part is what makes this better than just a plain old image classifier. For a non-text example: let’s say you’re scanning a receipt with a camera. The Transformer needs to tell if that patch is a $20 number, a logo, or a line item. The attention mechanism lets it “look” at other patches across the image—like, the patch of “$20” next to the patch that says “total” to confirm what that number means. That’s way more flexible than a CNN, which works by focusing on local blocks. For non-text data, attention lets the model connect dots that are nowhere near each other, which is a game changer.

Then there’s tabular data—you know, spreadsheets with columns like “age,” “income,” “purchase amount.” That’s another big one for us, especially for clients building recommendation engines. Text isn’t involved here at all, just numbers and categories. How do we handle that? We turn each column’s value into an embedding, too. For numbers, we can normalize them to a 0-1 range and map them to a vector. For categories, like “country” or “customer type,” we use a learned embedding table—think of it like a lookup table where every unique category gets its own vector. For example, “United States” might be a vector that places it close to “Canada” in the model’s space, and far from “Japan” so it doesn’t mix up purchasing patterns from different regions. Then we stack all these column embeddings into a sequence of tokens, and the attention layer can learn how different columns relate to each other. Like, if you’re building a recommendation model, it might notice that someone’s “age” and “purchase amount” are more strongly related than their “country” and “favorite product”—something a traditional model might miss.

Wait, let’s not forget time series data. That’s non-text too—think sensor readings from a factory machine, stock prices, or even heart rate data. All of these are sequences of numbers over time. Transformers work great here because they’re built for sequences, same as text. How? Each time step (like every minute of sensor data) becomes a token. But time series has a specific quirk: order matters. A heart rate spike at time 10 is different from the same spike at time 20, so we add positional embeddings to each token. That’s a key point people sleep on for non-text data. For text, positional embeddings tell the model “this token comes before that one.” For time series, it’s the same idea—telling the model “this data point happened first” so it doesn’t mix up cause and effect. We’ve had a client use this for predictive maintenance on heavy machinery; their Transformer could spot a subtle pattern in vibration data 3 weeks before a part failed, which old models missed because they couldn’t track long-range dependencies as well.

Now, let’s talk about the messy part—when non-text data is mixed with text. That’s super common too, right? Like a product page that has a description (text), an image (non-text), and a price (tabular). How does the Transformer handle that? It just makes different types of tokens, all in the same sequence. For the product page example, you’d have text tokens from the description, image patch tokens from the product photo, and a price embedding token. Then the attention mechanism can cross-reference them: it can look at the image patch of a jacket, the text that says “waterproof,” and the $99 price to learn that water-resistant outerwear priced around $100 is a popular category. That’s way more powerful than if you tried to split these into separate models—Transformer does it all in one.

As a Transformer supplier, a lot of our work is tweaking these non-text pipelines because default Transformers (like BERT or GPT) are built for text. Most people don’t realize that you can’t just plug an image into a vanilla Transformer and get good results. We adjust the tokenization, the embedding layers, and even the attention heads to work with whatever non-text data you’ve got. For example, if a client is working with medical data—like ultrasound images plus patient notes—we optimize the model to pay extra attention to cross links between the ultrasound patches and the relevant parts of the patient’s note, instead of random connections. That’s not something you can do out of the box; it’s customized for their specific data type.

Wait, let’s clear up a common myth: Transformers don’t “understand” non-text data. They just represent it as numerical vectors that let the attention mechanism find patterns. It’s all math, same as text embeddings. The attention is what drives the learning, regardless of what the input vectors represent. That’s why it’s so versatile—you can use it for almost any sequence or grid data, as long as you can turn it into a sequence of tokens with meaningful embeddings.

I’ve seen clients come to us thinking they need a whole new model for non-text tasks, only to realize that adjusting the Transformer’s input layers is enough. For example, a small retail client tried using a CNN for their image-based product search, but it was missing 15% of search results because it couldn’t connect similar products that looked a little different. We swapped out their CNN for a Transformer with image patch tokens, and within a month, their search accuracy went up 28%. The difference? The Transformer’s attention could match a blue shirt with a slightly darker blue shirt, even if the CNN saw them as totally different.

Another example: time series for energy grid management. A utility company used traditional models to predict demand, but they struggled with unusual events like holidays or heatwaves. We built a Transformer that takes time series tokens of past demand, plus embeddings for events like “July 4th” or “100°F day.” The attention layer learns to link those event embeddings to demand spikes, so the predictions are way more accurate during those times. That’s non-text, mixed with categorical data, all handled by the same Transformer.

Now, let’s get to the pitfalls. Non-text data can be messy, and that trips up a lot of new users. For tabular data, if you have missing values, you can’t just skip them—you have to turn them into a special embedding for “missing” so the model knows that entry was incomplete. For images, lighting or angle changes can make patches look totally different, so your embeddings need to be robust to that. For time series, if your data has irregular time steps (like a sensor that reads every 5 minutes sometimes, every hour other times), you need to add positional embeddings that account for the time between steps, not just the order. As a Transformer supplier, we help clients fix these issues before they derail their projects—because non-text data’s quirks are where most people run into problems.

So wrapping this up: Transformers handle non-text data by using three core steps, no fancy magic required. First, convert the non-text input into a sequence of tokens, whether that’s image patches, time steps, tabular columns, or category values. Second, assign each token a meaningful embedding that captures the data’s properties, plus positional embeddings to keep order (when it matters). Third, let the Transformer’s attention mechanism learn patterns by connecting tokens across the entire input, regardless of what type of data they are. That’s it.

If you’re working with non-text data—whether it’s images, time series, spreadsheets, or a mix—don’t limit yourself to old models that struggle with long-range connections or cross-modal links. Transformers are built for this, and as a Transformer supplier, we can help you customize the model to fit your specific use case, no matter how “untext-like” your data is. Don’t waste time building separate models for each data type—reach out to get the conversation started on how we can build something tailored for you.

Custom Machines References
Vaswani, A., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems
Dosovitskiy, A., et al. (2020). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale
Carion, N., et al. (2020). End-to-End Object Detection with Transformers
Wu, N., et al. (2021). Time Series Transformer: Unified Architecture and Empirical Studies for Forecasting and Classification


Wuxi Haifei Intelligent Equipment Co., Limited
Wuxi Haifei Intelligent Equipment Co., Limited is well-known as one of the leading transformer manufacturers and suppliers in China. Please rest assured to buy high quality transformer made in China here from our factory. For price consultation, contact us.
Address: 28 Shuiyun Road, Yuecheng, Jiangyin, Jiangsu Province, China. 214404
E-mail: WD03@busbarwelder.com
WebSite: https://www.busbarwelder.com/