Terminal
Terminal/Skill
Writes Python data processing and machine learning code that runs on BigQuery using the BigQuery DataFrames library instead of SQL.
By Google
Works with
npx skills add google/skills --skill bigquery-bigframesBigFrames is a Python library that lets you take advantage of BigQuery data processing by using familiar Python APIs.
Stay in the Cloud: Perform data cleaning, transformation, and analysis via BigFrames methods to leverage BigQuery's scale rather than downloading data.
Prefer partial ordering mode: Enable partial ordering mode right after importing BigFrames. This speeds up data processing significantly by relaxing row-sequence constraints.
import bigframes.pandas as bpd
bpd.options.bigquery.ordering_mode = 'partial'
Use peek() for data preview: Use peek(n) to preview data instead of
head(n). peek(n) randomly samples n rows and is significantly faster.
head(n) returns rows in strict order and fails in partial ordering mode
unless the DataFrame has been explicitly sorted.
Avoid materializing data locally: Methods like to_pandas() download all
data to client memory, bypassing BigQuery’s distributed computation and
risking Out of Memory (OOM) errors. Do not materialize data locally unless:
Prefer Dataframe API over SQL queries: Do not write raw SQL queries via
read_gbq() if a DataFrame/Series method achieves the same result, as it
breaks the Pandas abstraction and prevents lazy query execution.
Accessors over UDFs/Lambdas:
df.col.str.*, df.col.dt.*) instead of
remote User Defined Functions (UDFs). UDFs require extra resources and
time to deploy.Series.map() or DataFrame.apply(). These
methods do not accept functions without udf or remote_function
decorators.# Avoid:
df["upper"] = df["name"].map(lambda x: x.upper())
# Prefer:
df["upper"] = df["name"].str.upper()
Schema Verification: Do not assume the schema of intermediate outputs.
Proactively verify schemas using .dtypes and inspect sample records using
display() with .peek().
Visualization: Plot directly from the BigFrames DataFrame/Series when
possible. BigFrames is compatible with Matplotlib and Seaborn. If direct
plotting fails, use the .plot accessor. If the dataset is too large to plot,
aggregate or sample the data before calling
.to_pandas() to plot locally.
bigframes.bigquery.ml package: Do not use Scikit-learn or other ML
libraries with BigQuery DataFrames. Standard Scikit-learn models require
bringing data into local client memory, whereas bigframes.bigquery.ml
delegates training directly to BigQuery's scalable ML engine. Import functions
from bigframes.bigquery.ml.The BigFrames ML package (bigframes.ml) is a legacy package that mimics the
scikit-learn API but is no longer recommended for new projects. Only use this
package if the user explicitly requests BigFrames ML.
bigframes.ml instead of bigframes.bigquery.ml.predict() method always returns a DataFrame containing both predictions
and features, rather than a single series of predictions.random_state: Do not pass a random_state argument when
instantiating BigFrames ML models, as this parameter is not supported in the
BigFrames ML package.OneHotEncoder or StandardScaler unless
explicitly requested, as scaling is handled automatically.GridSearchCV or RandomizedSearchCV.bigframes.ml.forecasting.transform() method. Use predict()
instead.model.to_gbq(). To load a
persisted model, use bpd.read_gbq_model().