after first response from api chatbotify input field becoming disabled

import axios from "axios";
import { React } from "react";
import ChatBot from "react-chatbotify";
const ChatBotify = () => {
async function run(params) {
const authToken = localStorage.getItem("authToken");
const payload = {
userinput: params.userInput,
history: { quote: "14235436" }
};
const url = "urls";
const config = {
headers: {
"Content-Type": "application/json",
Authorization: Bearer ${authToken}
}
};
try {
const response = await axios.post(url, payload, config);
  if (response.data) {
    params.streamMessage(response.data.bot_response);
  } else {
    console.error("Unexpected response structure:", response);
    return "Unexpected response from server.";
  }
} catch (error) {
  console.error("Error during Axios request:", error);
  return "An error occurred while processing your request.";
}

}
const flow = {
start: {
message: "Hello, I am  ChatBot, How Can I Help You!",
path: "model_loop"
},
model_loop: {
message: async (params) => {
await run(params);
},
//   chatDisabled: false,
path: "model_loop"
}
};
const themes = [
// { id: "solid_purple_haze", version: "0.1.0" },
// { id: "minimal_midnight", version: "0.1.0" }
];
return (
<ChatBot
//   themes={themes}
settings={
{
audio: { disabled: false },
       voice: { disabled: false },

    }
  }
  flow={flow}
/>

);
};
export default ChatBotify;

this is urgent if anyone got the clue plz tell

@tjtanjin @team plz help me for this

Hey @ajaysingh7623, I tried the following on the playground (you can copy and paste it there as well):
const MyChatBot = () => {
    async function run(params) {
        const authToken = localStorage.getItem("authToken");
        const payload = {
            userinput: params.userInput,
            history: {
                quote: "14235436"
            }
        };
        const url = "urls";
        const config = {
            headers: {}
        }
        try {
            const response = await fetch('https://jsonplaceholder.typicode.com/todos/1')

            if (response.data) {
                params.streamMessage(response.data.bot_response);
            } else {
                console.error("Unexpected response structure:", response);
                return "Unexpected response from server.";
            }
        } catch (error) {
            console.error("Error during Axios request:", error);
            return "An error occurred while processing your request.";
        }
    }
    let count = 0;
    const flow = {
        start: {
            message: "Hello, I am ChatBot, How Can I Help You!",
            path: "model_loop"
        },
        model_loop: {
            message: async (params) => {
                return await run(params);
            },
            // chatDisabled: false,
            path: "model_loop"
        }
    };
    return ( <
        ChatBot settings = {
            {
                general: {
                    embedded: true
                },
                chatHistory: {
                    storageKey: "example_smart_conversation"
                }
            }
        }
        flow = {
            flow
        }
        themes={[]}
        />
    );
};

render( < MyChatBot / > )

This quite closely resembles what you have but I cannot reproduce the issue - can you verify your setup (with your urls) on the playground to see if you face it there as well? You can also reach out on discord if it's urgent - easier and faster to communicate there than via github issues.

@tjtanjin now also it is not working for me. facing same issue

@ajaysingh7623 Hmm that is very strange, the playground should be consistent for both of us. Could you take a similar video for the playground as you did for the earlier comment so that I can compare the visual discrepancies if any?

@tjtanjin i am using  "react-chatbotify": "^2.0.0-beta.1", version. and browser google chrome and operating system windows 10

You can bump up to beta.2, but I doubt that'll solve the issue. Issue with .trim() seems to suggest that within message processing, there's a certain piece of logic that expected a string but got something else. Can you check the bot response type and try removing params.streamMessage to return just a normal string to see if that's the issue?

@tjtanjin

response.data.bot_response is string type only
but if i remove params.streamMessage it will through same error and did not return anything in chatbotify screen as i attach in screenshot same error also coming.

@ajaysingh7623 Hmm let's try this incrementally and see where it starts to break:

Remove the function inside message and directly return a string instead.
If step 1 works, re-add the function but remove all current logic inside the function and just return a string.
If step 2 works, re-add the existing logic but mock the api response with a hard-coded string.
If step 3 works, run the existing code (this is where it currently breaks).

You can try the above and see at which step it starts to break.

@tjtanjin thank you that problem is solved by removing async and await from run call.
i have one issue audio is not working.

@tjtanjin thank you that problem is solved by removing async and await from run call.
i have one issue audio is not working code is same

Hmm double check that audio is not disabled and it's toggled on. Also note that audio can only start playing after the user interacts with the page. There's also a live audio example that you can refer to on the documentation page.

@tjtanjin  this is possible onlick of url in page it get redirected ?

Quote created D24TGYVF5-01, with transaction ID<PHONE_NUMBER>1, use this link to open the quote (https://cpq-itg.ext.hp.com/page/q/D24TGYVF5-01)

@tjtanjin  this is possible onlick of url in page it get redirected ?

Quote created D24TGYVF5-01, with transaction ID<PHONE_NUMBER>1, use this link to open the quote (https://cpq-itg.ext.hp.com/page/q/D24TGYVF5-01)

Yup that's achievable via the dangerouslySetInnerHtml option under botBubble that allows you to have html within the chat messages. You're responsible for sanitizing the input though!

@tjtanjin  this is possible onlick of url in page it get redirected ?

Quote created D24TGYVF5-01, with transaction ID<PHONE_NUMBER>1, use this link to open the quote (https://cpq-itg.ext.hp.com/page/q/D24TGYVF5-01)

Yup that's achievable via the dangerouslySetInnerHtml option under botBubble that allows you to have html within the chat messages. You're responsible for sanitizing the input though!

can you please provide the example or page so, i can refer to it.

@ajaysingh7623
The botBubble properties can be found here. There's also no tapToPlay property within audio so you can remove that (you may find the search bar at the top of the documentation useful for navigating around).
Does the audio example work for you?

@ajaysingh7623
The botBubble properties can be found here. There's also no tapToPlay property within audio so you can remove that (you may find the search bar at the top of the documentation useful for navigating around).
Does the audio example work for you?

@tjtanjin i removed tapToPlay attribute then also not working

@tjtanjin  this is possible onlick of url in page it get redirected ?

Quote created D24TGYVF5-01, with transaction ID<PHONE_NUMBER>1, use this link to open the quote (https://cpq-itg.ext.hp.com/page/q/D24TGYVF5-01)

Yup that's achievable via the dangerouslySetInnerHtml option under botBubble that allows you to have html within the chat messages. You're responsible for sanitizing the input though!

@tjtanjin can you please provide the example or page so, i can refer to it.

but this will provide bold ,italic text
not like href link so we can click in it and move to that page @tjtanjin

@tjtanjin  this is possible onlick of url in page it get redirected ?

Quote created D24TGYVF5-01, with transaction ID<PHONE_NUMBER>1, use this link to open the quote (https://cpq-itg.ext.hp.com/page/q/D24TGYVF5-01)

Yup that's achievable via the dangerouslySetInnerHtml option under botBubble that allows you to have html within the chat messages. You're responsible for sanitizing the input though!

@tjtanjin can you please provide the example or page so, i can refer to it.

but this will provide bold ,italic text not like href link so we can click in it and move to that page @tjtanjin

It also provides the anchor element <a> for you to make the link clickable. By default it's being treated as a string because the chatbot isn't making assumptions about the output desired.

@ajaysingh7623
The botBubble properties can be found here. There's also no tapToPlay property within audio so you can remove that (you may find the search bar at the top of the documentation useful for navigating around).
Does the audio example work for you?

@tjtanjin i removed tapToPlay attribute then also not working

Is the audio example working for you on the documentation website?

@ajaysingh7623
The botBubble properties can be found here. There's also no tapToPlay property within audio so you can remove that (you may find the search bar at the top of the documentation useful for navigating around).
Does the audio example work for you?

@tjtanjin i removed tapToPlay attribute then also not working

Is the audio example working for you on the documentation website?

yes it is working in documentation
but same thing i used in my code it is not working

@ajaysingh7623 As a sanity check have you checked local audio and have you tried a different device or browser? If the documentation website example is working then there's a good chance that it's a local issue.

Thank you @tjtanjin for :-after first response from api chatbotify input field becoming disabled


savefig cuts away luminosity label

I noticed an issue with the function savefig when used with plot_mc. The code below produces the plots plot_mc.pdf and compare_data_mc.pdf. However, for plot_mc.pdf the luminosity label is cut away in the pdf, while it works fine with compare_data_mc.pdf.
from plothist.generate_dummy_data import generate_dummy_data
from plothist import make_hist
from plothist import plot_mc
from plothist import compare_data_mc
from plothist import savefig
from plothist import add_luminosity

df = generate_dummy_data()
hist = make_hist(df['variable_0'],bins=50)

fig, ax = plot_mc(
    mc_hist_list = [hist],
    xlabel='variable_0'
)

add_luminosity(ax=ax, is_data=False, preliminary=True)
savefig(fig,'plot_mc.pdf')

fig, ax, ax_comparison = compare_data_mc(
    data_hist = hist,
    mc_hist_list = [hist],
    xlabel='variable_0'
)

add_luminosity(ax=ax, is_data=False, preliminary=True)
savefig(fig,'compare_data_mc.pdf')

compare_data_mc.pdf
plot_mc.pdf

Thank you for your feedback!
This happens because the figsize is not large enough to have every element display. To fix this, you have the argument new_figsize to increase the size of the saved figure. This will only enlarge the figsize and not the size of the subplots.
For you example, you can do:
savefig(fig, 'plot_mc.pdf', new_figsize=(6, 4.2)) # Increase the height of the saved fig

The savefig function is still experimental, and using fig.savefig(path, bbox_inches="tight") is highly recommended.


Type handling (serialization) improvements

A follow up of #3453, this PR simplifies the interface used to serialize objects and makes it extremely easy to serialize arbitrary objects using either Protobuf or Gson. TypeHandlingIntegrationTest shows how easy it is to serialize an arbitrary generic object with supertype references in fields (the test linked below includes both serialization and deserialization via Protobuf):
https://github.com/MovingBlocks/Terasology/blob/c9fd8934282be2d0bf57968a6e1c9ecc8cffc5b0/engine-tests/src/test/java/org/terasology/persistence/typeHandling/TypeHandlingIntegrationTest.java#L87-L111
For context, the class being serialized in the above test is this (where Animal is an abstract class with two subclasses):
    private static class SomeClass<T> {
        private T data;
        private List<T> list = Lists.newArrayList();
        private Set<Animal> animals = Sets.newHashSet();
    }

Changes

Remove DeserializationContext
Add factories for type handlers (Rect2f, Rect2i, Region3i) previously relying on DeserializationContext.deserializeAs which look up the type handlers they need
Remove SerializationContext.create(T, Class<? extends T>)
Rename SerializationContext to PersistedDataSerializer
Rename PersistedDataSerializer.createXYZ methods to serializeXYZ
Add ArrayTypeHandler (+ Factory)

TODO

[x] Add null handling in every TypeHandler -- null/default value handling takes place in ObjectFieldMapTypeHandler but still look for a better way to easily handle null values without having to if(value == null) { return serializer.serializeNull(); } in every single type handler
[ ] Further simplify external Gson and Protobuf serialization workflow
[x] ~Load types from name via a ClassLoader(s) in RuntimeDelegatingTypeHandlerto stay within the sandbox~ To be handled in a later PR
[ ] Rebase onto develop after #3453 is merged

Uh oh, something went wrong with the build. Need to check on that

Hooray Jenkins reported success with all tests good!

Hooray Jenkins reported success with all tests good!

Hooray Jenkins reported success with all tests good!

Hooray Jenkins reported success with all tests good!

Hooray Jenkins reported success with all tests good!

Tested this out to continue efforts earlier on Slack - I still get an error on JoshariasSurvival :-(
22:03:14.397 [main] INFO  o.t.e.prefab.internal.PrefabFormat - Attempting to deserialize prefab ManualLabor:ToolAssemblingBaseProcessPart with inputs [/ManualLabor/assets/prefabs/processDefinitions/ToolAssemblingBaseProcessPart.prefab]
22:03:14.499 [main] ERROR o.terasology.engine.TerasologyEngine - Uncaught exception, attempting clean game shutdown
java.lang.IllegalStateException: Data is not an array: bytes: ""

	at org.terasology.persistence.typeHandling.protobuf.ProtobufPersistedData.getAsValueArray(ProtobufPersistedData.java:394)
	at org.terasology.persistence.typeHandling.protobuf.ProtobufPersistedData.iterator(ProtobufPersistedData.java:401)
	at org.terasology.persistence.typeHandling.coreTypes.CollectionTypeHandler.deserialize(CollectionTypeHandler.java:56)
	at org.terasology.persistence.typeHandling.TypeHandler.deserializeOrNull(TypeHandler.java:68)
	at org.terasology.persistence.typeHandling.Serializer.deserializeOnto(Serializer.java:96)
	at org.terasology.persistence.typeHandling.Serializer.deserializeOnto(Serializer.java:153)
	at org.terasology.persistence.serializers.ComponentSerializer.deserializeOnto(ComponentSerializer.java:197)
	at org.terasology.persistence.serializers.ComponentSerializer.deserialize(ComponentSerializer.java:114)
	at org.terasology.persistence.serializers.PrefabSerializer.applyComponentChanges(PrefabSerializer.java:164)
	at org.terasology.persistence.serializers.PrefabSerializer.deserialize(PrefabSerializer.java:136)
	at org.terasology.persistence.serializers.PrefabSerializer.deserialize(PrefabSerializer.java:116)
	at org.terasology.entitySystem.prefab.internal.PrefabFormat.load(PrefabFormat.java:55)
	at org.terasology.entitySystem.prefab.internal.PrefabFormat.load(PrefabFormat.java:36)
	at org.terasology.assets.module.UnloadedAssetData$AssetSourceResolver.load(UnloadedAssetData.java:292)
	at org.terasology.assets.module.UnloadedAssetData.load(UnloadedAssetData.java:180)
	at org.terasology.assets.module.ModuleAssetDataProducer.getAssetData(ModuleAssetDataProducer.java:228)
	at org.terasology.assets.AssetType.lambda$reload$2(AssetType.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.terasology.assets.AssetType.reload(AssetType.java:357)
	at org.terasology.assets.AssetType.getNormalAsset(AssetType.java:386)
	at org.terasology.assets.AssetType.getAsset(AssetType.java:262)
	at org.terasology.assets.management.AssetManager.getAsset(AssetManager.java:223)
	at org.terasology.engine.modes.loadProcesses.LoadPrefabs.step(LoadPrefabs.java:42)
	at org.terasology.engine.modes.StateLoading.update(StateLoading.java:240)
	at org.terasology.engine.TerasologyEngine.tick(TerasologyEngine.java:458)
	at org.terasology.engine.TerasologyEngine.mainLoop(TerasologyEngine.java:421)
	at org.terasology.engine.TerasologyEngine.run(TerasologyEngine.java:397)
	at org.terasology.engine.Terasology.main(Terasology.java:154)
22:03:14.499 [main] INFO  o.terasology.engine.TerasologyEngine - Shutting down Terasology...

Conflict resolution is easy enough, but like #3449 this seems to break at least one module - I suspect the same small list as https://github.com/MovingBlocks/Terasology/pull/3449#pullrequestreview-141360142 but am finding it a little trickier to rush through broken files to confirm the next broken module in the list. Could you please pull those modules and see if you can create fix PRs for them please, if more than just Dialogs is broken?

[ ] Dialogs (used by L&S)
[ ] DynamicCities
[ ] LightAndShadow - GSOC sensitive
[ ] MasterOfOreon - GSOC sensitive
[ ] Tasks (used by L&S)
[ ] TutorialDynamicCities

Last time around I tested with Light & Shadow and MasterOfOreon since they cover a ton of dependencies. I neglected testing JS which was silly. I also checked out Record & Replay for that sake. Did both some single player, multiplayer, and reloading a saved world.

Hooray Jenkins reported success with all tests good!

Can confirm the latest commit seems to fix JoshariasSurvival 👍
Would love to merge but can't sink the extra time into also fixing modules right now. Maybe tomorrow during some quiet time as we don't need to test JS right away - although it certainly would nice to also have this one thrown in!

Made associated module issues, although I can't assign them to any of the people I'm pinging as they aren't public members of the Terasology org - am happy to help make that happen if needed :-)
Or rather, I started making issues and testing things, but I'm finding additional stuff so I won't be able to finish tonight. It tends to be easier to fix modules in a mega-workspace then build again to see the next error. Hard to leave broken modules in place and then dance around them trying to test other things. There are runtime errors in addition to compile time, for instance GooeysQuests fully compiles but also crashes at runtime. Maybe we can make an initial set of PRs then try with those merged locally.
@eviltak I pinged you in the issues and am also listing them here as an overview, but without public org membership I can't actually assign the issues. GitHub is funny like that.
Modules that do not compile
Easy enough to provoke: Try to compile the things! :-)

[ ] https://github.com/Terasology/Dialogs/issues/8
[ ] https://github.com/Terasology/Tasks/issues/7
[ ] https://github.com/Terasology/DynamicCities/issues/33
[ ] https://github.com/Terasology/LightAndShadow/issues/74
[ ] https://github.com/Terasology/MasterOfOreon/issues/29

Indirectly affected as dependencies
Can't test without modules above fixed

ManualLabor
QuestExamples
TutorialDynamicCities (probably fine)
MedievalCities (probably fine - just assets)

Compile OK but issues found with this PR

[ ] https://github.com/Terasology/GooeyDefence/issues/30 - crash on auto-save with certain activities present
[ ] https://github.com/Terasology/GooeysQuests/issues/31 - crash on world init

Note: merged into a parking branch for merging at some point via #3490


The order of series is not maintained when updating a chart

I have a chart with multiple series which are toggled outside the chart. When I toggle an element of an then on again it is placed at the end of the dataseries array instead of its intended location. This is a problem for me as it messes up the z-index and also the legend is incorrectly ordered.
I have already looked at the code and will create a PR for this issue.

Thanks for the PR I will review soon! do you have an example of the trouble you were experiencing?


Migrate from Rmd and/or .ipynb

This tasks consists of moving a notebook from Rmd and/or jupyter notebook to quarto and render it.
If you feel brave enough, you can try to migrate from other solutions (i.e. blogdown or jupyterbook) to quarto.
Read this description and form a group with anyone interested in doing this task (minimum 2 people -you can comment below and/or assign it to yourselves to show your interest).
Steps to reproduce:

Generate an output file either html, pdf, docx...) in your preferred layout (simple report, article, paper's template... ) from quarto either by
2. Editing/adding quarto's yaml's config into an existing ipynb or rmd file
3. Changing file extension to qmd and making edits to adapt the code following quarto's syntax

Expected outputs

Article: source code (qmd file) and output (html, pdf, docx...). This can be within this repo or linked to an external repo (in that case, link it when documenting the process).
Documentation (see template). Pay special attention to describing the steps and limitations (if any!)

References
Guides

VS Code/notebook editor
Getting started

I can't self assign because I don't have the right access to this repo, but I'd love to work on this 🚀

I'm also interested in this. Not sure if it come under the "brave" category, but I'd be interesting in converting a xaringan presentation as one option.

I'd love that! Xaringan has such pretty themes and I'd love to know if something comparable is already available for Quarto.


Fix GraphiQL component trying to use undefined properties

TypeError: Cannot read property 'storage' of undefined
TypeError: this.storageGet is not a function

whoaa silly ones


Support default method

I will propose to support default method on following interface method:

Cache#getReadWriteLock
DataSourceFactory#setProperties
KeyGenerator#processBefore
KeyGenerator#processAfter
ProxyFactory#setProperties
DatabaseIdProvider#setProperties
Interceptor#plugin
Interceptor#setProperties
ObjectFactory#setProperties
Transaction#getTimeout
TransactionFactory#setProperties

As apply this changes, source code becomes a little slim.
Adding as follows: (2018-04-11)

SqlSession#*
SqlSessionFactory#*

Fix conflict as follows: (2018-06-23)

DefaultSqlSession

WDYT?

Hi @kazuki43zoo ,
I'm sorry about the late reply 🐢
I just took a quick look at the methods you mentioned.
These look OK to me.

Cache#getReadWriteLock
DatabaseIdProvider#setProperties
ProxyFactory#setProperties
Interceptor#plugin
Interceptor#setProperties
ObjectFactory#setProperties
TransactionFactory#setProperties

I'm not sure about these:

DataSourceFactory#setProperties : No-op seems unlikely
KeyGenerator#processBefore : See below
KeyGenerator#processAfter : One of these methods should be implemented, so defining no-op default might not be a good idea.
Transaction#getTimeout : I'm just not sure. :D

@harawata
I agree with opinion! I've applied your comment. And I will consider to modify a SqlSession and SqlSessionFactory by another PR.


[GAE-Java] grape-graphx support multiple executors per host.

Currently, we only support the setting that spark launches only  one executor one host, when multiple small executors are launched on one host, the program works abnormally.

Fixed in https://github.com/alibaba/GraphScope/pull/2185


Do not rewrite file if it already exists

Closes: https://github.com/ggrossetie/asciidoctor-kroki/issues/431
Do not rewrite file if it already exists in Node environment.
This matches the behavior implemented in Ruby:

https://github.com/ggrossetie/asciidoctor-kroki/pull/370

Seems this issue got fixed in this commit: https://github.com/asciidoctor/asciidoctor-kroki/commit/9eb59c3832d09febbb8928e6bac8357ab63b1be6

All good. I understand. Thanks for all your work in the asciidoctor.js ecosystem.


setHasOptionsMenu behavior

On older devices where AB is handled by ABS (Android 4.0.4) the menu is created right away when calling setHasOptionsMenu(true) from fragment's onCreate method causing possible NPE if the we rely on objects in onCreateOptionsMenu that are created later in onCreate.


Linking to other containers

The docs state that linking to other containers can be done with additional arguments.
https://github.com/dokku-alt/dokku-alt#linking-to-other-containers
The dokku containers don't have persistent names, so how should I reference the container when linking it?

This is a duplicate of https://github.com/dokku-alt/dokku-alt/issues/226


Add a theme

Put a theme in your blog.. It's ugly as s*** with this default one.

It is already planned on the roadmap.

Só you can reference the commit in this issue


[10.x] Retain $request->request InputBag type

When we create the Illuminate request, we set the $request->request property to the value of the $request->json property.
Symfony is expecting $request->request to be an InputBag when invoking $request->getPayload(). See:
https://github.com/symfony/symfony/blob/42d9d7cd64c5b9897a2113c12d76c75a8d31ad9c/src/Symfony/Component/HttpFoundation/Request.php#L1514
The $request->json property is an instance of ParameterBag. This is the parent class of InputBag.
So we are setting $request->request to an instance of ParameterBag instead of InputBag which is then breaking due to the return type on Request::getPayload()
Side note
It is possible that we should make $request->json an instance of InputBag to make things consistent, however this is a quick fix to fix the framework asap.
fixes #47834

Can this get back-ported to the 9.x branch as well?

See: https://github.com/laravel/framework/pull/47840


Component detail not loading

When I click on the details of my component, I expect a detail view of my component, however a modal pops up with following console errors:
user-bundle.js:1394 Warning: You are manually calling a React.PropTypes validation function for the `visible` prop on `Modal`. This is deprecated and will not work in the next major version. You may be seeing this warning due to a third-party PropTypes library. See https://fb.me/react-warning-dont-call-proptypes for details.

Here is my component:
// src/components/Avatar/index.js
import React from "react";

const Avatar = () => (

This will become an avatar

);

Avatar.propTypes = {
};

export default Avatar;

and my webpack config:
const path = require("path");
const CarteBlanche = require("carte-blanche");
const HtmlWebpackPlugin = require("html-webpack-plugin");

const CarteBlancheConfig = new CarteBlanche({
  componentRoot: "./src/components",
  dest: "components",
});

const HtmlWebpackPluginConfig = new HtmlWebpackPlugin({
  inject: true,
  template: path.join(__dirname, "./src/index.html"),
});

module.exports = {
  entry: "./src/index.js",
  output: {
    path: path.join(__dirname, "/dist"),
    filename: "bundle.js",
  },
  watch: true,

  module: {
    loaders: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        loader: "babel-loader",
        query: {
          presets: ["react", "es2015"],
        },
      },
    ],
  },

  resolve: {
    extensions: ["", ".js"],
  },

  plugins: [
    CarteBlancheConfig,
    HtmlWebpackPluginConfig,
  ],
};

Can you tell me if I'm doing something wrong?

Seeing the same, it seems that there is something wrong with the css, because the Card is really on the page, but hidden by the modal

It is not on the page (you are seeing the front page), and due to the console message it is a JavaScript error... I am experiencing the same too, when i tried to pull carte blanche into an existing project...

Same here. The error is actually:
index.js:302 Generating metadata failed Error: getNodeFromInstance: Invalid argument.
    at invariant (invariant.js:38)
    at Object.getNodeFromInstance (ReactDOMComponentTree.js:156)
    at Object.findDOMNode (findDOMNode.js:50)
    at Constructor._getDOMTarget (velocity-component.js:128)
    at Constructor.runAnimation (velocity-component.js:107)
    at Constructor.componentDidMount (velocity-component.js:66)
    at ReactCompositeComponentWrapper.invokeComponentDidMountWithTimer (ReactCompositeComponent.js:63)
    at CallbackQueue.notifyAll (CallbackQueue.js:66)
    at ReactReconcileTransaction.close (ReactReconcileTransaction.js:79)
    at ReactReconcileTransaction.closeAll (Transaction.js:202)


Attach issue with service process and PrivateTmp setting

I have encountered the issue when the service is configured with:
PrivateTmp=true

This setting allows to sets up a new file systemnamespace for the executed processes and mounts a private /tmp directory inside it, that is not shared by processes outside of the namespace.
We can easily encounter the issue with it when our library is presented in the /tmp e.g. in /tmp/libscope-web/libscope.so
flowchart LR
  libscope.so-->/tmp
  subgraph Host
    /tmp
    subgraph service
        `/tmp`
    end
  end

In inject function we are passing the library path in FIRST_ARG_REG. Since the process operates on a private /tmp directory it will not find our library so the dlopen -> inject -> attach will fail.
error: dlopen() failed, library could not be injected

See example configuration for memcache service:
memcached.service

Additional information: I think we can distinguish this case by checking the mount ns.
The current mechanism used in the AppScope to distinguish processes running on the host and on the container based on different PID namespace, but the example mentioned above is that process can work in the same PID namespace but in a different mount namespace.

Test instructions
Docker test
Create a container in the same PID namespace as the host and start htop process
docker run --privileged --pid=host --name testcont --rm -it ubuntu:22.04 bash
apt update -y
apt install htop netcat -y

Attach from host
pgrep htop ### retrieve PID of htop
sudo SCOPE_CRIBL_ENABLE=false ldscope -a <htop_pid>

Observe that there is no message like below
error: failed to find libc in target process

Verify that data is flowing in the container
docker exec -it testcont bash
nc -l -p 9109

Service test
On the host environment install memcached
apt install memached

verify if Memcached is running as a service
systemctl status memcached

You should see something like:
     Loaded: loaded (/lib/systemd/system/memcached.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-10-24 13:50:42 CEST; 1min 22s ago
       Docs: man:memcached(1)
   Main PID: 590408 (memcached)
      Tasks: 10 (limit: 38111)
     Memory: 1.4M
     CGroup: /system.slice/memcached.service
             └─590408 /usr/bin/memcached -m 64 -p 11211 -u memcache -l <IP_ADDRESS> -P /var/run/memcached/

Verify if PrivateTmp is set to True
grep PrivateTmp /lib/systemd/system/memcached.service
PrivateTmp=true

Attach to memached process
sudo SCOPE_CRIBL_ENABLE=false ldscope -a <memcached_pid>
#e.g. the memached_pid is 590408 in example above
nc -l -p 9109

See that there is no error message and data is flowing.


All tests fail with Attempt to free invalid pointer 0x82799f060

Failure:
$ python3.9 test_arit.py
src/tcmalloc.cc:333] Attempt to free invalid pointer 0x82799f060
Abort trap

Stack trace:
#0  thr_kill () at thr_kill.S:4
#1  0x000000080072d104 in __raise (s=s@entry=6) at /disk-samsung/freebsd-src/lib/libc/gen/raise.c:52
#2  0x00000008007dddc9 in abort () at /disk-samsung/freebsd-src/lib/libc/stdlib/abort.c:67
#3  0x0000000805025591 in tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) ()
   from /usr/local/lib/libtcmalloc.so.4
#4  0x0000000805021b85 in ?? () from /usr/local/lib/libtcmalloc.so.4
#5  0x00000008006668a0 in _thr_mutexattr_destroy (attr=0x7fffffffaba0) at /disk-samsung/freebsd-src/lib/libthr/thread/thr_mutexattr.c:180
#6  0x00000008052de489 in std::__1::__libcpp_recursive_mutex_init (__m=<optimized out>) at /disk-samsung/freebsd-src/contrib/llvm-project/libcxx/include/__threading_support:268
#7  std::__1::recursive_mutex::recursive_mutex (this=0x80414a140) at /disk-samsung/freebsd-src/contrib/llvm-project/libcxx/src/mutex.cpp:56
#8  0x0000000802d95f2f in ?? () from /usr/local/lib/libsymengine.so.0.9
#9  0x0000000802d7a744 in ?? () from /usr/local/lib/libsymengine.so.0.9
#10 0x0000000802d7a6b5 in ?? () from /usr/local/lib/libsymengine.so.0.9
#11 0x0000000802cb1d61 in ?? () from /usr/local/lib/libsymengine.so.0.9
#12 0x0000000802ccc161 in ?? () from /usr/local/lib/libsymengine.so.0.9
#13 0x000000080020e1bd in objlist_call_init (list=list@entry=0x7fffffffb610, lockstate=lockstate@entry=0x7fffffffb590) at /disk-samsung/freebsd-src/libexec/rtld-elf/rtld.c:3141
#14 0x0000000800212966 in dlopen_object (name=name@entry=0x801850750 "/usr/local/lib/python3.9/site-packages/symengine/lib/symengine_wrapper.cpython-39.so", fd=fd@entry=-1,
    refobj=<optimized out>, lo_flags=<optimized out>, mode=mode@entry=2, lockstate=0x7fffffffb590, lockstate@entry=0x0) at /disk-samsung/freebsd-src/libexec/rtld-elf/rtld.c:3889
#15 0x000000080020f1ae in rtld_dlopen (name=0x801850750 "/usr/local/lib/python3.9/site-packages/symengine/lib/symengine_wrapper.cpython-39.so", fd=-1, mode=<optimized out>)
    at /disk-samsung/freebsd-src/libexec/rtld-elf/rtld.c:3749
#16 0x0000000800506b21 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#17 0x00000008004d1a6f in ?? () from /usr/local/lib/libpython3.9.so.1.0
#18 0x00000008004d13b6 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#19 0x000000080040f457 in ?? () from /usr/local/lib/libpython3.9.so.1.0

symengine.py-0.9.2
symengine-0.9.0
Python-3.9
clang-14
FreeBSD 13.1

Thanks for reporting this nasty bug. Since this doesn't show up in the CI on Mac and Linux I guess this is related to FreeBSD or possible clang-14 or the llvm c++ standard library.
In the stacktrace we don't see from where in libsymengine we are calling. Could you possibly compile with debug flags to get a better view of the trace? Do you know if it is failing on the very first test, some other test or an all tests?

#0  thr_kill () at thr_kill.S:4
#1  0x0000000800802104 in __raise (s=s@entry=6) at /disk-samsung/freebsd-src/lib/libc/gen/raise.c:52
#2  0x00000008008b2dc9 in abort () at /disk-samsung/freebsd-src/lib/libc/stdlib/abort.c:67
#3  0x0000000805d8f591 in tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) ()
   from /usr/local/lib/libtcmalloc.so.4
#4  0x0000000805d8bb85 in ?? () from /usr/local/lib/libtcmalloc.so.4
#5  0x000000080073b8a0 in _thr_mutexattr_destroy (attr=0x7ffffffec7e0) at /disk-samsung/freebsd-src/lib/libthr/thread/thr_mutexattr.c:180
#6  0x0000000806048489 in std::__1::__libcpp_recursive_mutex_init (__m=<optimized out>) at /disk-samsung/freebsd-src/contrib/llvm-project/libcxx/include/__threading_support:268
#7  std::__1::recursive_mutex::recursive_mutex (this=0x804fb9b60 <getManagedStaticMutex()::m>) at /disk-samsung/freebsd-src/contrib/llvm-project/libcxx/src/mutex.cpp:56
#8  0x0000000803b6873f in llvm::ManagedStaticBase::RegisterManagedStatic(void* (*)(), void (*)(void*)) const () from /usr/local/lib/libsymengine.so.0.9
#9  0x0000000803b4cfe4 in llvm::cl::OptionCategory::registerCategory() () from /usr/local/lib/libsymengine.so.0.9
#10 0x0000000803b4cf55 in llvm::cl::getGeneralCategory() () from /usr/local/lib/libsymengine.so.0.9
#11 0x0000000803a84601 in llvm::cl::opt<bool, false, llvm::cl::parser<bool> >::opt<char [42], llvm::cl::desc, llvm::cl::OptionHidden>(char const (&) [42], llvm::cl::desc const&, llvm::cl::OptionHidden const&) () from /usr/local/lib/libsymengine.so.0.9
#12 0x0000000803a9ea01 in _GLOBAL__sub_I_X86AsmParser.cpp () from /usr/local/lib/libsymengine.so.0.9
#13 0x000000080020e1bd in objlist_call_init (list=list@entry=0x7ffffffed250, lockstate=lockstate@entry=0x7ffffffed1d0) at /disk-samsung/freebsd-src/libexec/rtld-elf/rtld.c:3141
#14 0x0000000800212966 in dlopen_object (name=name@entry=0x8018579d0 "/usr/local/lib/python3.9/site-packages/symengine/lib/symengine_wrapper.cpython-39.so", fd=fd@entry=-1,
    refobj=<optimized out>, lo_flags=<optimized out>, mode=mode@entry=2, lockstate=0x7ffffffed1d0, lockstate@entry=0x0) at /disk-samsung/freebsd-src/libexec/rtld-elf/rtld.c:3889
#15 0x000000080020f1ae in rtld_dlopen (name=0x8018579d0 "/usr/local/lib/python3.9/site-packages/symengine/lib/symengine_wrapper.cpython-39.so", fd=-1, mode=<optimized out>)
    at /disk-samsung/freebsd-src/libexec/rtld-elf/rtld.c:3749
#16 0x00000008005bb418 in _PyImport_FindSharedFuncptr (prefix=0x8002b271c "PyInit", shortname=0x8018bf890 "symengine_wrapper",
    pathname=0x8018579d0 "/usr/local/lib/python3.9/site-packages/symengine/lib/symengine_wrapper.cpython-39.so", fp=0x0) at ./Python/dynload_shlib.c:100
#17 0x0000000800570e6e in _PyImport_LoadDynamicModuleWithSpec (spec=0x8018c9d30, fp=0x0) at ./Python/importdl.c:137
#18 0x00000008005709ab in _imp_create_dynamic_impl (module=0x801415680, spec=0x8018c9d30, file=0x0) at Python/import.c:2302
#19 0x000000080056feae in _imp_create_dynamic (module=0x801415680, args=0x8018c9898, nargs=1) at Python/clinic/import.c.h:330
#20 0x000000080045b316 in cfunction_vectorcall_FASTCALL (func=0x801428b30, args=0x8018c9898, nargsf=1, kwnames=0x0) at Objects/methodobject.c:430
#21 0x00000008004000e9 in PyVectorcall_Call (callable=0x801428b30, tuple=0x8018c9880, kwargs=0x8018cb8c0) at Objects/call.c:231
#22 0x00000008004001ec in _PyObject_Call (tstate=0x800e4b000, callable=0x801428b30, args=0x8018c9880, kwargs=0x8018cb8c0) at Objects/call.c:266
#23 0x00000008004002e2 in PyObject_Call (callable=0x801428b30, args=0x8018c9880, kwargs=0x8018cb8c0) at Objects/call.c:293
#24 0x000000080053b661 in do_call_core (tstate=0x800e4b000, func=0x801428b30, callargs=0x8018c9880, kwdict=0x8018cb8c0) at Python/ceval.c:5097
#25 0x0000000800537fce in _PyEval_EvalFrameDefault (tstate=0x800e4b000, f=0x801529200, throwflag=0) at Python/ceval.c:3582
#26 0x000000080052c04f in _PyEval_EvalFrame (tstate=0x800e4b000, f=0x801529200, throwflag=0) at ./Include/internal/pycore_ceval.h:40
#27 0x000000080053c4ec in _PyEval_EvalCode (tstate=0x800e4b000, _co=0x801418c90, globals=0x801425f00, locals=0x0, args=0x80180a1d0, argcount=2, kwnames=0x0, kwargs=0x80180a1e0, kwcount=0,
    kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x801415a30, qualname=0x801415a30) at Python/ceval.c:4329
#28 0x000000080040062d in _PyFunction_Vectorcall (func=0x80142c3a0, stack=0x80180a1d0, nargsf=9223372036854775810, kwnames=0x0) at Objects/call.c:396
#29 0x000000080053de79 in _PyObject_VectorcallTstate (tstate=0x800e4b000, callable=0x80142c3a0, args=0x80180a1d0, nargsf=9223372036854775810, kwnames=0x0)
    at ./Include/cpython/abstract.h:118
#30 0x000000080053b2fa in PyObject_Vectorcall (callable=0x80142c3a0, args=0x80180a1d0, nargsf=9223372036854775810, kwnames=0x0) at ./Include/cpython/abstract.h:127
#31 0x000000080053b3e9 in call_function (tstate=0x800e4b000, pp_stack=0x7fffffff0148, oparg=2, kwnames=0x0) at Python/ceval.c:5077
#32 0x00000008005379d8 in _PyEval_EvalFrameDefault (tstate=0x800e4b000, f=0x80180a040, throwflag=0) at Python/ceval.c:3489
#33 0x000000080040209f in _PyEval_EvalFrame (tstate=0x800e4b000, f=0x80180a040, throwflag=0) at ./Include/internal/pycore_ceval.h:40
#34 0x000000080040070b in function_code_fastcall (tstate=0x800e4b000, co=0x801445b30, args=0x800bc9a40, nargs=2, globals=0x8014354c0) at Objects/call.c:330
#35 0x0000000800400425 in _PyFunction_Vectorcall (func=0x80146e700, stack=0x800bc9a30, nargsf=9223372036854775810, kwnames=0x0) at Objects/call.c:367
#36 0x000000080053de79 in _PyObject_VectorcallTstate (tstate=0x800e4b000, callable=0x80146e700, args=0x800bc9a30, nargsf=9223372036854775810, kwnames=0x0)
    at ./Include/cpython/abstract.h:118
#37 0x000000080053b2fa in PyObject_Vectorcall (callable=0x80146e700, args=0x800bc9a30, nargsf=9223372036854775810, kwnames=0x0) at ./Include/cpython/abstract.h:127
#38 0x000000080053b3e9 in call_function (tstate=0x800e4b000, pp_stack=0x7fffffff1448, oparg=2, kwnames=0x0) at Python/ceval.c:5077
#39 0x0000000800537a21 in _PyEval_EvalFrameDefault (tstate=0x800e4b000, f=0x800bc98b0, throwflag=0) at Python/ceval.c:3506
#40 0x000000080040209f in _PyEval_EvalFrame (tstate=0x800e4b000, f=0x800bc98b0, throwflag=0) at ./Include/internal/pycore_ceval.h:40

This happens inside of dlopen, and is related to mutexes.

When LLVM option was turned OFF the stack changed:
#1  0x0000000800802104 in __raise (s=s@entry=6) at /disk-samsung/freebsd-src/lib/libc/gen/raise.c:52
#2  0x00000008008b2dc9 in abort () at /disk-samsung/freebsd-src/lib/libc/stdlib/abort.c:67
#3  0x0000000803e92591 in tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) ()
   from /usr/local/lib/libtcmalloc.so.4
#4  0x0000000803e8eb85 in ?? () from /usr/local/lib/libtcmalloc.so.4
#5  0x000000080073cdd9 in _thr_rwlock_destroy (rwlock=<optimized out>) at /disk-samsung/freebsd-src/lib/libthr/thread/thr_rwlock.c:136
#6  0x0000000849653b43 in CRYPTO_THREAD_lock_free (lock=0x6a5a6) at /disk-samsung/freebsd-src/crypto/openssl/crypto/threads_pthread.c:107
#7  0x00000008496d29d2 in DSO_free (dso=0x800e5b050) at /disk-samsung/freebsd-src/crypto/openssl/crypto/dso/dso_lib.c:92
#8  0x0000000849650bb1 in ossl_init_load_crypto_nodelete () at /disk-samsung/freebsd-src/crypto/openssl/crypto/init.c:204
#9  ossl_init_load_crypto_nodelete_ossl_ () at /disk-samsung/freebsd-src/crypto/openssl/crypto/init.c:159
#10 0x000000080073bab3 in _thr_once (once_control=0x849823360, init_routine=0x849650b70 <ossl_init_load_crypto_nodelete_ossl_>) at /disk-samsung/freebsd-src/lib/libthr/thread/thr_once.c:98
#11 0x0000000849653b69 in CRYPTO_THREAD_run_once (once=0x6a5a6, init=0x6) at /disk-samsung/freebsd-src/crypto/openssl/crypto/threads_pthread.c:118
#12 0x0000000849650622 in OPENSSL_init_crypto (opts=opts@entry=8, settings=settings@entry=0x0) at /disk-samsung/freebsd-src/crypto/openssl/crypto/init.c:649
#13 0x0000000849741b8d in EVP_MD_do_all (fn=0x849498430 <_openssl_hash_name_mapper>, arg=0x7ffffffad5a8) at /disk-samsung/freebsd-src/crypto/openssl/crypto/evp/names.c:162
#14 0x0000000849491ee5 in hashlib_md_meth_names (module=0x848ffd4f0) at Modules/_hashopenssl.c:1900
#15 0x0000000849491b97 in PyInit__hashlib () at Modules/_hashopenssl.c:2283
#16 0x0000000800570f31 in _PyImport_LoadDynamicModuleWithSpec (spec=0x848ffcc70, fp=0x0) at ./Python/importdl.c:167
#17 0x00000008005709ab in _imp_create_dynamic_impl (module=0x801415680, spec=0x848ffcc70, file=0x0) at Python/import.c:2302
#18 0x000000080056feae in _imp_create_dynamic (module=0x801415680, args=0x848ffcbc8, nargs=1) at Python/clinic/import.c.h:330
#19 0x000000080045b316 in cfunction_vectorcall_FASTCALL (func=0x801428b30, args=0x848ffcbc8, nargsf=1, kwnames=0x0) at Objects/methodobject.c:430
#20 0x00000008004000e9 in PyVectorcall_Call (callable=0x801428b30, tuple=0x848ffcbb0, kwargs=0x848d3bfc0) at Objects/call.c:231
#21 0x00000008004001ec in _PyObject_Call (tstate=0x800e4b000, callable=0x801428b30, args=0x848ffcbb0, kwargs=0x848d3bfc0) at Objects/call.c:266

When I ran the python testcase with HEAPCHECK=normal LD_PRELOAD=/usr/local/lib/libtcmalloc.so the error disappeared.

Will also run with Valgrind.

Valgrind didn't find anything specific in SymEngine.
But turning off google perftools caused failures to disappear. IMO this means that this is a memory corruption. The behavior changes with memory layout changes.

Memory corruption could have occurred earlier and it crashed when the Python interpreter called some OpenSSL code later.

Yes, true. Very challenging to debug in that case. We don't know for sure that symengine is causing it. Could also be compiler, libraries, Python or operating system. Perhaps changing some of the other moving parts could give some clues.

I turned off the TCMALLOC in SymEngine in the FreeBSD port as a workaround. perftools changes memory allocation and this triggers this issue. But the problem is most likely somewhere in SymEngine IMO.

@yurivict thanks for the report and for debugging this. Is SymEngine compiled in Debug mode with all checks enabled? Is the bug still there?
Things can fail in Release mode if we have a bug in the code, but usually in Debug build our asserts will catch it.
If you think the bug is in symengine, then we need to figure out how to reproduce it reliably on your machine. Once we have that, then we need to "bisect" (manually removing parts in the large reproducer to make the reproduce smaller) until we figure out what is going on.

Is SymEngine compiled in Debug mode with all checks enabled? Is the bug still there?

Yes, Debug code falls the same way.

If you think the bug is in symengine, then we need to figure out how to reproduce it reliably on your machine.

It can be reliably reproduced on FreeBSD. The math/symengine port should be built with TCMALLOC=ON, and then math/py-symengine reliably fails.

Thanks, perfect. I think it's a bug, so let's fix it.
If you have time to work on this a bit, what you can do is to try to "minimize" the reproducer. For example can you reproduce the bug without py-symengine? It would be great to eliminate Python somehow. Do you think the bug is in the Python wrappers or in the C++ symengine?

Do you think the bug is in the Python wrappers or in the C++ symengine?

I don't know.


Erlang/Elixir

The top decription here says 'Erlang' while Elixir is now prefered.

Elixir can "seamlessly" call Erlang functions, so it's more accurate to say Erlang as you can still use Rustler for Erlang code, even if you're transitively using Elixir somewhere in there.

I mean that the description text and the text in the README bite each other.
Plus, its not clear that Elixir is supported, reading the description line.


ExternalTexture for custom renderer

I read this issue here about making a custom renderer for playing video ; https://github.com/google-ar/sceneform-android-sdk/issues/169
Now ExternalTexture is out and google has released a code sample playing a video through it : https://github.com/google-ar/sceneform-android-sdk/blob/master/samples/chromakeyvideo/app/src/main/java/com/google/ar/sceneform/samples/chromakeyvideo/ChromaKeyVideoActivity.java
My problem with this is that I don't just want to render a video, I have a complex view mixing images, and videos, and other things (like a small openGL game). So I need to render this view on a sceneform node. Could it be possible to render that view into an ExternalTexture ? The only problem would be that, contrary to ViewRenderable, the user interactions with my android view would not be managed.
But is it theoretically possible ? Do you see another way to achieve what I'm looking for ?
If rendering an android view to an ExternalTexture is possible, how would you manage the user interactions with that view ? I was thinking about adding a ViewRenderable view at exact same anchor and same size/orientation, and passing the touch events to the android view, but is this possible ?

That is exactly what ViewRenderable does, it renders the View into an external texture. Is there a reason you cannot use ViewRenderable directly? What type of View are you using exactly?

Thx very much for the reply. The view I would like to render is a kind of rich media/game engine. It contains in its hierarchy several ImageViews and several TextureViews rendering videos through MediaPlayer. All views are mixed, an ImageView can hide a piece of a TextureView, and reversally.
As far as I can know, the TextureViews is not supported by ViewRenderable? I'm sorry I can't test it right now because my phone is not supported by ARCore, that will be fixed soon.
My initial question was, considering that TextureViews are not supported by ViewRenderable, would it be possible to render a whole android view that contains ImageViews and TextureViews into an ExternalTexture ? But the answer is probably "yes but TextureViews won't be rendered" because you just said "That is exactly what ViewRenderable does" .
Since then I tried a few more things. To solve the problem of TextureViews not being supported by ViewRenderable, I've been working on a CustomVideoView. It's a ViewGroup which contains in its hierarchy an ImageView and a TextureView (hidden, out of bounds). The TextureView renders the video with MediaPlayer, and 30 times per seconds I use the TextureView.getBitmap method to copy its content into the ImageView. That looks scary and dirty, but it works and the performance is better than I expected. But this is dirty, if you see a better way to do this, let me know. Anyway, I can't test it with ARCore right now, but for me this has good chances to work fine.
The other possibility I had was to re-write this rich media/game engine using ARCore ExternalTexture for playing videos and displaying images. But that's a very painful path for dev and maintenance.
Let me know it's not clear.
Also, one last question. I plan to adapt a couple of small OpenGL games (GLSurfaceViews) to ARCore. I just want to show the games as they are through a ViewRenderable in 2D plane anchored in the 3D AR space. Once again, GLSurfaceViews are not supported, so my plan was first to make the games use TextureViews instead of GLSurfaceViews (which seems not to be very difficult to do), and use the same hack I dead for my CustomVideoView. Let me know if you see better alternatives.

You are right that TextureView and SurfaceView are not supported by ViewRenderable. TextureView could be made to work in theory though, but that would require changes in Sceneform.
You don't need to change your games to use TextureView. Whether you are using a GLSurfaceView or a TextureView you are rendering into a Surface in the end. All you need is the ability to set the rendering Surface in the game. You can check the UI helper we built for Filament (Sceneform's renderer) that makes it easy to use either a SurfaceView or a TextureView.

Support of TextureView would be so great indeed.
OK I get the idea for the gl views, the idea would be to render the gl directly to the ExternalSurface. I'll look into the UI Helper class. Thank you very much.


ckb: 0.2.8 -> 2018-01-01

Motivation for this change
The upstream package has not had a release since May 2017, but the master branch has had a lot of updates since, including support for new devices. I therefore thought it made sense to update the nixos package to the latest master commit.
I don't know if this is common practise though, so if we'd rather follow releases I could as easily make a ckb-unstable package.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

IMHO, it's much better for end users if upstream makes a new release. So personally I'd spend my energy on helping upstream (or applying gentle pressure) rather than creating separate nixpkgs package and/or updating to unreleased software.


Play Again Button

I'm making an issue about this, to make sure you don't forget my suggestion and when you have time, you can do it if you want

Added in ae3978c


[BUG] Data corrupted in the timestamp field to 1970-01-01 19:45:30.000 after subsequent upsert run

Tips before filing an issue

Have you gone through our FAQs?

Join the mailing list to engage in conversations and get faster support at<EMAIL_ADDRESS>

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
I am using hudi as a part of Glue jobs to manage data mutations in our data lake. The only modification that is being applied during the ETL from raw to bronze (where hudi is involved) is introduction of timestamp field that transforms date in seconds field from raw (ie<PHONE_NUMBER> -> 2020-10-10 02:45:30.000). I have noticed that for some entities, after subsequent ingestions, the timestamp field value becomes corrupted (even for the records that are not present in the update). I managed to isolate the bulk_insert as a source of the issue. The timestamp field gets reset to 1970-01-01 19:45:30.000. Just to note, when insert is being used in place of bulk_insert for the initial ingest, the issue is not occurring.
To Reproduce
Steps to reproduce the behavior:
Unfortunately it is pretty tricky to reproduce and it is only happening to 2 types of entities out of 16 we are using.
We are moving stripe data into the data lake and the entities that do get issues are (https://stripe.com/docs/api/invoices and https://stripe.com/docs/api/customers

Create AWS Glue ETL job with hudi config
hudi_options = {          "hoodie.table.create.schema": "default",          "hoodie.table.name": tableName,          "hoodie.datasource.write.recordkey.field": "id",          "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",          "hoodie.datasource.write.partitionpath.field": "field_1, field_2",          "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",          "hoodie.datasource.hive_sync.partition_fields": "field_1, field_2",          'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS',          'hoodie.cleaner.commits.retained': 10,          "hoodie.datasource.write.table.name": tableName,          "hoodie.datasource.write.operation": "bulk_insert",          "hoodie.parquet.compression.codec": "snappy",          "hoodie.datasource.hive_sync.enable": "true",          "hoodie.datasource.hive_sync.use_jdbc": "false",          'hoodie.datasource.hive_sync.support_timestamp': "true",          "hoodie.datasource.write.precombine.field": "lastupdate",          "hoodie.datasource.hive_sync.database": f"{database_name}",          "hoodie.datasource.hive_sync.table": tableName,      }
The transformation of the raw data looks as following:
sparkDF = dfc.select(list(dfc.keys())[0]).toDF() transformed = sparkDF.select("*", from_unixtime(col("created")).cast("timestamp").alias("created_ts")) resolvechoice4 = DynamicFrame.fromDF(transformed, glueContext, "transformedDF") return DynamicFrameCollection({"CustomTransform0": resolvechoice4}, glueContext)

Write data to data lake
sparkDF.write.format("hudi").options(**hudi_options).mode("overwrite").save(          basePath      )

Update the previously mentioned hudi config with write operation to become upsert. Update spark mode to append

Trigger Glue job again with updates

Expected behavior
I expect the data to remain consistent and the fields to represent the relevant info
Environment Description

Hudi version : 0.9.0

Spark version : 3.1

Hive version : Using Glue data catalogue as hive metastore

Hadoop version : N/A

Glue version: 3

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) :

Additional context
Add any other context about the problem here.
Stacktrace
There is no error

yeah. We recently fixed the issue here. Can you try out the patch and let us know if it resolves the issue.

Closing this one out since we know the root cause and have a solution. Feel free to re-open if you have more questions. would be happy to help.

I tried the patch from https://github.com/apache/hudi/pull/4203 but it still does not work for me.
The only difference is that I upserts data using SQL MERGE INTO statement.
I run my job in the AWS Glue - first job execution dumps data to S3 and creates table in Glue Data Catalog(BULK_INSERT). Then I add property(to serde properties) to Glue table hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true, so the SQL command can read this property. But when I'm running upsert job, it still corrupts all the timestamps within the files that are being updated.
@nsivabalan Could you help me with it, please?

Did you set the property when you did BULK_INSERT?  guess thats where the config is effective or rather the fix was made if I am not wrong. Basically can you ensure you set the config value in all operations (bulk_insert and uspert) and see what happens.

I've tried it for BULK_INSERT and UPSERT as well, bot nothing works for me.
I prepared sample test job of that so you can reproduce it or just review it (I hope I just misconfigured it :grinning:).
Upsert operation corrupts timestamps, no matter if I set this option or not -  I tested it using DataFrame writer and SQL MERGE INTO.
AWS Glue 2.0(Spark 2.4)
Hudi 0.10.1
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
import org.apache.hudi.hive.MultiPartKeysValueExtractor
import org.apache.hudi.hive.ddl.HiveSyncMode
import org.apache.hudi.hive.util.ConfigUtils
import org.apache.hudi.keygen.ComplexKeyGenerator
import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.functions.{current_timestamp, expr, lit}
import org.apache.spark.sql.hudi.HoodieOptionConfig.{SQL_KEY_PRECOMBINE_FIELD, SQL_KEY_TABLE_PRIMARY_KEY, SQL_KEY_TABLE_TYPE}
import org.apache.spark.sql.hudi.{HoodieOptionConfig, HoodieSparkSessionExtension}
import org.apache.spark.sql.{SaveMode, SparkSession}

import scala.collection.JavaConverters.mapAsJavaMapConverter

case class HudiTableOptions(tableName: String,
                            databaseName: String,
                            operationType: String,
                            partitionColumns: List[String],
                            recordKeyColumns: List[String],
                            preCombineColumn: String) {

  def hudiTableOptions = Map(
    TABLE_TYPE.key() -> COW_TABLE_TYPE_OPT_VAL,
    OPERATION.key() -> operationType,
    TBL_NAME.key() -> tableName,
    RECORDKEY_FIELD.key() -> recordKeyColumns.mkString(","),
    PARTITIONPATH_FIELD.key() -> partitionColumns.mkString(","),
    KEYGENERATOR_CLASS_NAME.key() -> classOf[ComplexKeyGenerator].getName,
    PRECOMBINE_FIELD.key() -> preCombineColumn,
    URL_ENCODE_PARTITIONING.key() -> false.toString,
    KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.key() -> true.toString
  )

  def hiveTableProperties = Map(
    SQL_KEY_TABLE_TYPE.sqlKeyName -> HoodieOptionConfig.SQL_VALUE_TABLE_TYPE_COW,
    SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName -> hudiTableOptions(RECORDKEY_FIELD.key()),
    SQL_KEY_PRECOMBINE_FIELD.sqlKeyName -> hudiTableOptions(PRECOMBINE_FIELD.key())
  )

  def hiveTableOptions = Map(
    HIVE_SYNC_MODE.key() -> HiveSyncMode.HMS.name(),
    HIVE_SYNC_ENABLED.key() -> true.toString,
    HIVE_DATABASE.key() -> databaseName,
    HIVE_TABLE.key() -> hudiTableOptions(TBL_NAME.key()),
    HIVE_PARTITION_FIELDS.key() -> hudiTableOptions(PARTITIONPATH_FIELD.key()),
    HIVE_PARTITION_EXTRACTOR_CLASS.key() -> classOf[MultiPartKeysValueExtractor].getName,
    HIVE_STYLE_PARTITIONING.key() -> true.toString,
    HIVE_SUPPORT_TIMESTAMP_TYPE.key() -> true.toString,
    HIVE_TABLE_SERDE_PROPERTIES.key() -> ConfigUtils.configToString(hiveTableProperties.asJava)
  )

  def writerOptions: Map[String, String] = hudiTableOptions ++ hiveTableOptions
}

object GlueApp {
  def main(args: Array[String]): Unit = {
    Logger.getRootLogger.setLevel(Level.WARN)
    Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)
    Logger.getLogger("org.apache.hudi").setLevel(Level.INFO)

    val spark: SparkSession = SparkSession.builder()
      .appName("Test")
      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .config("hive.metastore.glue.catalogid", "****")
      .withExtensions(new HoodieSparkSessionExtension())
      .enableHiveSupport()
      .getOrCreate()

    import spark.implicits._

    val tableName = "hudi_test"
    val databaseName = "test_database"
    val path = "s3://s3bucketname/hudi-test/"
    val tableOptions = HudiTableOptions(tableName, databaseName, BULK_INSERT_OPERATION_OPT_VAL, List("year"), List("id"), "ts")

    val dataFrameForBulkInsert = Range(0, 10000).toDF("id")
      .withColumn("year", lit(2022))
      .withColumn("ts", current_timestamp() - expr("INTERVAL 100 DAYS"))
      .withColumn("other", current_timestamp() - expr("INTERVAL 150 DAYS"))

    val dataFrameForUpsert = Range(5000, 15000).toDF("id")
      .withColumn("year", lit(2022))
      .withColumn("ts", current_timestamp() - expr("INTERVAL 10 DAYS"))
      .withColumn("other", current_timestamp() - expr("INTERVAL 50 DAYS"))

    // ------------------------ BULK INSERT ------------------------------------------
    dataFrameForBulkInsert.write
      .format("org.apache.hudi")
      .options(tableOptions.writerOptions)
      .mode(SaveMode.Overwrite)
      .save(path)
    // -----------------------------------------------------------------------------

    Thread.sleep(10 * 1000)

    // --------------------- UPSERT Spark DataFrame Writer --------------------------
//    dataFrameForUpsert
//      .write
//      .format("org.apache.hudi")
//      .options(tableOptions.copy(operationType = UPSERT_OPERATION_OPT_VAL).writerOptions)
//      .mode(SaveMode.Append)
//      .save(path)
    // ------------------------------------------------------------------------------

    // --------------------- UPSERT SQL MERGE INTO-----------------------------------
    val mergeIntoStatement =
      s"""MERGE INTO $databaseName.$tableName AS t
         | USING source_data_set s
         | ON t.id=s.id
         | WHEN MATCHED THEN UPDATE SET *
         | WHEN NOT MATCHED THEN INSERT *
         |""".stripMargin
    dataFrameForUpsert.createTempView("source_data_set")
    spark.sql(mergeIntoStatement)
    // -------------------------------------------------------------------------------
  }
}

@YannByron : Looks like the timestamp consistency fix we(guess you and sagar) landed does not work well. Do you think you can follow up here please?

@denys-tyshetskyy @rafcis02
This issue, i think, is a bug for Bulk_Insert and hoodie.datasource.write.row.writer.enable.
But for now, you can disable hoodie.datasource.write.row.writer.enable when bulk_insert. That will work well. And we'll fix it asap.

@denys-tyshetskyy @rafcis02
update this: you can set hoodie.parquet.outputtimestamptype to TIMESTAMP_MICROS  when you bulk_insert datas. It'll work nicely.

Thanks a lot @YannByron, hoodie.parquet.outputtimestamptype works perfectly

thanks Yann 👍
@rafcis02 @denys-tyshetskyy : if you are good, can we close the issue.

Thanks guys, feel free to close

Hello @nsivabalan,  I have the same problem.
The solution presented here solves the problem for the bulk insert but the upsert remains the same.
In my case I have a job that on the 1st load does bulk_insert and then inserts data on new loads, all these new loads have dates from 1970.
I'll leave my configs attached:
Initial Load :
commonConfig = {'className' : 'org.apache.hudi',
'hoodie.datasource.hive_sync.use_jdbc':'false',
'hoodie.datasource.write.precombine.field': 'updated_date',
'hoodie.datasource.write.recordkey.field': primaryKey,
'hoodie.table.name': tableName,
'hoodie.consistency.check.enabled': 'true',
'hoodie.datasource.hive_sync.database': database,
'hoodie.datasource.hive_sync.table': tableName,
'hoodie.datasource.hive_sync.enable': 'true',
"hoodie.datasource.hive_sync.support_timestamp": "true",
'path': targetPath
}
partitionDataConfig = {'hoodie.datasource.write.partitionpath.field': partitionKey,
'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.hive_sync.partition_fields': partitionKey
}
initLoadConfig = {'hoodie.bulkinsert.shuffle.parallelism': 68,
'hoodie.datasource.write.operation': 'bulk_insert',
'hoodie.datasource.write.payload.class': 'org.apache.hudi.common.model.EmptyHoodieRecordPayload',
'hoodie.datasource.write.partitions.to.delete': targetPath,
'hoodie.parquet.outputtimestamptype' : 'TIMESTAMP_MICROS'
}
combinedConf = {**commonConfig, **partitionDataConfig, **initLoadConfig}
glueContext.write_dynamic_frame.from_options(frame = DataSource0, connection_type = "marketplace.spark", connection_options = combinedConf)
Incremental Load :
commonConfig = {'className' : 'org.apache.hudi',
'hoodie.datasource.hive_sync.use_jdbc':'false',
'hoodie.datasource.write.precombine.field': 'updated_date',
'hoodie.datasource.write.recordkey.field': primaryKey,
'hoodie.table.name': tableName,
'hoodie.consistency.check.enabled': 'true',
'hoodie.datasource.hive_sync.database': database,
'hoodie.datasource.hive_sync.table': tableName,
'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.support_timestamp': 'true',
'path': targetPath
}
partitionDataConfig = {'hoodie.datasource.write.partitionpath.field': partitionKey,
'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.hive_sync.partition_fields': partitionKey
}
incrementalConfig = {'hoodie.upsert.shuffle.parallelism': 20,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS',
'hoodie.parquet.outputtimestamptype' : 'TIMESTAMP_MICROS'
}
combinedConf = {**commonConfig, **partitionDataConfig, **incrementalConfig}
DataSink0 = glueContext.write_dynamic_frame.from_options(frame = DynamicFrame.fromDF(DataSourceDF0, glueContext, "initDf"), connection_type = "marketplace.spark", connection_options = combinedConf)
Would it be possible for you to help me?
thanks in advance

I second that @roSimoes, I have exactly same use case: initial bulk_insert big table and then upsert new loads into it.
Timestamps of upserted rows end up corrupted to 1970-01-01 aswell. Have you found any solution?

https://github.com/apache/hudi/issues/5469
The 0.11.0 version fixed it.


The cli has drifted badly out of sync with functionality changes

Especially after the db backend migration, I wouldn't be surprised to find that none of the registered commands work at all. They need updated to be useful again.

I tore the useless cli code out in #9. Right now there's nothing.


Python 3.6 support is broken

[admin@takserver ~]$ adsbxcot
Traceback (most recent call last):
  File "/usr/local/bin/adsbxcot", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/adsbxcot/commands.py", line 134, in cli
    loop = get_event_loop()
NameError: name 'get_event_loop' is not defined

Line 25 of commands.py reads:
    from asyncio import _get_running_loop as get_running_loop

See: https://github.com/ampledata/adsbxcot/blob/main/adsbxcot/commands.py#L25
It should read:
    from asyncio import get_event_loop as get_running_loop


Make specification creation and mixing easier, remove boilerplate from specifications

This is my take on the work done in the past weeks by @madig and @graphicore
This likely deprecates PR #1761

So, what should be done when I want to extend e.g. the ufo_sources specification with site-specific checks? Import ufo_sources.specification and add stuff to it?

Import ufo_sources.specification and add stuff to it

The reason not to do this is that you shouldn't alter other modules.
And auto discovery of specs and their contents is opt-in, with the highest level of automation when specifying spec_factory. I'll write docs these days :-)

I'm hitting a problem with spec_imports.
# -*- coding: utf-8 -*-

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

from fontbakery.callable import check
from fontbakery.callable import FontBakeryExpectedValue as ExpectedValue
from fontbakery.checkrunner import ERROR, FAIL, INFO, PASS, SKIP, WARN, Section

from fontbakery.specifications.ufo_sources import UFOSpec
from fontbakery.specifications.ufo_sources import specification as ufo_base_spec

spec_imports = ((".", ("shared_conditions")))

specification = UFOSpec(
    default_section=Section('Dalton Maag'),
    iterargs={'font': 'fonts'},
    derived_iterables={'ufo_fonts': ('ufo_font', True)})
specification.merge_specification(ufo_base_spec)

expected_check_ids = [
    "com.daltonmaag/check/ufolint",
    "com.daltonmaag/check/required-fields",
    "com.daltonmaag/check/recommended-fields",
    "com.daltonmaag/check/unnecessary-fields",
]

# ---

@check(
    id='com.daltonmaag/check/test',
    description="...")
def check_encoding(ufo_font):
  yield PASS, "..."

# ---

specification.auto_register(globals())

specification.test_expected_checks(expected_check_ids, exclusive=True)

Gets me this stack trace:
Traceback (most recent call last):
  File "C:\UsersLocal\nikolaus.waxweiler\Development\tmp\fontbakery-venv\Scripts\fontbakery-script.py", line 11, in <module>
    load_entry_point('fontbakery', 'console_scripts', 'fontbakery')()
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\cli.py", line 24, in main
    "fontbakery.commands." + subcommand_module, run_name='__main__')
  File "C:\Python27\Lib\runpy.py", line 192, in run_module
    fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\commands\check_specification.py", line 226, in <module>
    sys.exit(main())
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\commands\check_specification.py", line 169, in main
    specification = get_spec()
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\commands\check_specification.py", line 149, in get_spec
    imported = import_module(args.specification, package=None)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\specifications\damaa.py", line 40, in <module>
    specification.auto_register(globals())
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\checkrunner.py", line 1429, in auto_register
    all_items = list(symbol_table.values()) + self._load_spec_imports(symbol_table)
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\checkrunner.py", line 1397, in _load_spec_imports
    module = importlib.import_module(module_name, package=package)
  File "C:\Python27\Lib\importlib\__init__.py", line 38, in import_module
    return sys.modules[name]
KeyError: u'fontbakery.specifications.'

Taking out spec_imports and importing manually solves the issue...

I believe:
spec_imports = ((".", ("shared_conditions")))
should be:
spec_imports = ((".", ("shared_conditions",)),)
but maybe:
spec_imports = [(".", ["shared_conditions"])]
works better for you to see it. The python tuple literals syntax is complicated if you have just one entry.

just thinking, if you do:
spec_imports = ((".", ("shared_conditions", "ufo_sources")),)

you don't need to do:
specification.merge_specification(ufo_base_spec)

Okay, that seemed to work. Now I'm hitting something new:
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

from fontbakery.callable import check
from fontbakery.checkrunner import ERROR, FAIL, INFO, PASS, SKIP, WARN, Section

from fontbakery.specifications.ufo_sources import UFOSpec

spec_imports = ((".", (
    "ufo_sources",
    "shared_conditions",
)),)

specification = UFOSpec(
    default_section=Section('Dalton Maag'),
    iterargs={'font': 'fonts'},
    derived_iterables={'ufo_fonts': ('ufo_font', True)})

expected_check_ids = [
    "com.daltonmaag/check/ufolint",
    "com.daltonmaag/check/required-fields",
    "com.daltonmaag/check/recommended-fields",
    "com.daltonmaag/check/unnecessary-fields",
    "com.daltonmaag/check/test"
]

@check(id='com.daltonmaag/check/test', description="...")
def check_encoding(ufo_font):
  yield PASS, "..."

specification.auto_register(globals())

specification.test_expected_checks(expected_check_ids, exclusive=True)

Trace:
There is a total of 4 checks on <Section: Default>.
Traceback (most recent call last):
  File "C:\UsersLocal\nikolaus.waxweiler\Development\tmp\fontbakery-venv\Scripts\fontbakery-script.py", line 11, in <module>
    load_entry_point('fontbakery', 'console_scripts', 'fontbakery')()
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\cli.py", line 24, in main
    "fontbakery.commands." + subcommand_module, run_name='__main__')
  File "C:\Python27\Lib\runpy.py", line 192, in run_module
    fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\commands\check_specification.py", line 226, in <module>
    sys.exit(main())
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\commands\check_specification.py", line 169, in main
    specification = get_spec()
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\commands\check_specification.py", line 149, in get_spec
    imported = import_module(args.specification, package=None)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\specifications\damaa.py", line 39, in <module>
    specification.auto_register(globals())
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\checkrunner.py", line 1448, in auto_register
    self.merge_specification(specification, filter_func=filter_func)
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\checkrunner.py", line 1477, in merge_specification
    self._add_dict_to_namespace(ns_type, ns_dict)
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\checkrunner.py", line 907, in _add_dict_to_namespace
    self.add_to_namespace(type, key, value, force=getattr(value, 'force', False))
  File "c:\userslocal\nikolaus.waxweiler\development\fontbakery\lib\fontbakery\checkrunner.py", line 928, in add_to_namespace
    , type, value))
fontbakery.checkrunner.NamespaceError: Name "fonts" is already registered in "expected_values" (value: <FontBakeryExpectedValue:fonts>). Reqested registering in "expected_values" (value: <FontBakeryExpectedValue:fonts>).

You should probably file issues for these things. They can help other users in the future if they can find them in the issue tracker.
What's happening is:
shared_conditions is a FontSpec and that defines it's own fonts name as an FontBakeryExpectedValue in the specification namespace. the UFOSpec in ufo_sources does the same. Now, since both name definitions are not the exact same object (identical instance), this is a namespace clash. Which one is the right fonts item to pick?
In this case you obviously want to use the one of the ufo_sources names. And you should probably explicitly chose not to import the names of the FontsSpec of which the specification of shared_conditions is a type of. Either by only importing explicit names from the shared_conditions module:
spec_imports = (
    ".ufo_sources",
    ( ".shared_conditions", ('"condition1", "conditon2"'))
)

or by using the filter_func argument of specification.auto_register:
def filter_func(type, name_or_id, item):
  if name == "fonts":
    return False
  return True

specification.auto_register(globals(), filter_func=filter_func)

In general, I'd advise to be careful when mixing specifications with different purposes, it may be a too big gap semantically even when things have the same name. "Namespace-hygiene" will probably be a topic we'll always need some diligence.


An example where ForneyLab is used to estimate player skill ratings

Player skills are based on the outcomes of several matches, as in the Infer.NET example.
https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/matchup-app-infer-net
The results are good for the skill means, but the variances do not (yet) agree with the reference.
(Work done at Sioux LIME)

Hi @KeithWM , thanks for this contribution! It's really nice to see a version of this application implemented with ForneyLab.
The current PR seems to be opened from the master branch of your ForneyLab fork, I wasn't sure whether this was intentional. Instead I used your infer_ratings_example branch and cleaned up the demo some more. Because I don't have the authorization to push to your remote, I opened a new PR, #123 . Please let me know there whether you agree with my changes. I will close the current PR.

Thanks for your efforts! I've never collaborated using Github before, it seems I got confused about a few things. Your changes are fine, I don't care about the state of my fork, I will probably remove it altogether. I didn't initially even make a fork but just worked in a local copy of the main master.


Two Goals for Cavern Survival

double goal - double reward! :D

@nixel2007 If only!

Strange, I see it only once Popey.

Yeah, only one for me now.  Haven't seen a patch, must just have been a bug.  Strange that I saw it on both direct + normal.


pass arbitrary arguments to mpiexec via charmrun

Original issue: https://charm.cs.illinois.edu/redmine/issues/1302

It would potentially be useful to be able to pass arguments to mpiexec via the charmrun command line when necessary options aren't automatically determined from environment variables set by the queueing system.

Original date: 2018-06-21 21:08:27

Does ./charmrun ++mpiexec ++remote-shell "mpiexec --YourOptionsHere" work?

Original date: 2018-12-19 19:17:31

Yes, that does work.
Please add to the docs at http://charm.cs.illinois.edu/manuals/html/charm++/C.html#mpiexec

Original date: 2018-12-19 20:16:32

~~https://charm.cs.illinois.edu/gerrit/c/charm/+/4872~~ https://github.com/UIUC-PPL/charm/commit/b3629a4217341476de16a2956992a29954c513d0


ML - model source type page

Replacing card component with radio group card since it's deprecated
Added a label for when a model source is selected

Pull Request Test Coverage Report for Build 329706564

30 of 62 (48.39%)  changed or added relevant lines in 6 files are covered.
6 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.04%) to 39.705%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

extensions/machine-learning/src/views/viewBase.ts
1
3
33.33%

extensions/machine-learning/src/views/models/modelViewBase.ts
6
10
60.0%

extensions/machine-learning/src/views/models/modelSourcesComponent.ts
13
39
33.33%

Files with Coverage Reduction
New Missed Lines
%

extensions/machine-learning/src/views/models/modelSourcePage.ts
3
72.73%

extensions/machine-learning/src/views/models/modelSourcesComponent.ts
3
42.57%

Totals

Change from base Build 329688694:
-0.04%

Covered Lines:
20533

Relevant Lines:
47454

💛  - Coveralls

@llali I have minor changes to check in here for styles in modelSourcesComponent.ts


Idea for the logo [concept]

Great. :)
What about adding another slash to remember sed pattern, and perhaps use a monospace font?

An maybe put the S on the same baseline, it looks weird like this ;) Nice work !

This logo looks very ugly, this is just an idea of concept ;)


HBASE-28153 Upgrade zookeeper to a newer version (#5484)

Backport of #5475
(cherry picked from commit 0d04a6053a05525e0ba8fd6525c3af7f3412c9de)

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 41s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2.4 Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 20s
branch-2.4 passed

+1 :green_heart:
compile
4m 27s
branch-2.4 passed

+1 :green_heart:
spotless
0m 41s
branch has no errors when running spotless:check.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 13s
the patch passed

+1 :green_heart:
compile
4m 28s
the patch passed

+1 :green_heart:
javac
4m 28s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  4s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
16m 40s
Patch does not cause any errors with Hadoop 2.10.2 or 3.1.4 3.2.4 3.3.6.

+1 :green_heart:
spotless
0m 39s
patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 40s
The patch does not generate ASF License warnings.

35m 14s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5490

Optional Tests
dupname asflicense javac hadoopcheck spotless xml compile

uname
Linux e11bb6eeaa47 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.4 / 57fea961ac

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
77 (vs. ulimit of 30000)

modules
C: hbase-zookeeper hbase-server hbase-it . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 48s
Docker mode activated.

-0 :warning:
yetus
0m  4s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2.4 Compile Tests _

+0 :ok:
mvndep
0m 17s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 25s
branch-2.4 passed

+1 :green_heart:
compile
1m 39s
branch-2.4 passed

+1 :green_heart:
shadedjars
4m 18s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m 42s
branch-2.4 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 17s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 16s
the patch passed

+1 :green_heart:
compile
1m 41s
the patch passed

+1 :green_heart:
javac
1m 41s
the patch passed

+1 :green_heart:
shadedjars
4m 20s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m 39s
the patch passed

_ Other Tests _

-1 :x:
unit
230m 58s
root in the patch failed.

260m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5490

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 4aaa58408dd8 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.4 / 57fea961ac

Default Java
Eclipse Adoptium-11.0.17+8

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/testReport/

Max. process+thread count
8291 (vs. ulimit of 30000)

modules
C: hbase-zookeeper hbase-server hbase-it . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 20s
Docker mode activated.

-0 :warning:
yetus
0m  4s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2.4 Compile Tests _

+0 :ok:
mvndep
0m 18s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 20s
branch-2.4 passed

+1 :green_heart:
compile
1m 12s
branch-2.4 passed

+1 :green_heart:
shadedjars
4m 11s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m  1s
branch-2.4 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 54s
the patch passed

+1 :green_heart:
compile
1m 15s
the patch passed

+1 :green_heart:
javac
1m 15s
the patch passed

+1 :green_heart:
shadedjars
4m 15s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m  1s
the patch passed

_ Other Tests _

-1 :x:
unit
263m 19s
root in the patch failed.

292m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5490

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 3b85b7578f00 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.4 / 57fea961ac

Default Java
Temurin-1.8.0_352-b08

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/testReport/

Max. process+thread count
6125 (vs. ulimit of 30000)

modules
C: hbase-zookeeper hbase-server hbase-it . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 44s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2.4 Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 15s
branch-2.4 passed

+1 :green_heart:
compile
6m 22s
branch-2.4 passed

+1 :green_heart:
spotless
0m 55s
branch has no errors when running spotless:check.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 19s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 19s
the patch passed

+1 :green_heart:
compile
6m  9s
the patch passed

+1 :green_heart:
javac
6m  9s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  6s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
19m 53s
Patch does not cause any errors with Hadoop 2.10.2 or 3.1.4 3.2.4 3.3.6.

+1 :green_heart:
spotless
0m 43s
patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 34s
The patch does not generate ASF License warnings.

44m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5490

Optional Tests
dupname asflicense javac hadoopcheck spotless xml compile

uname
Linux 43f9c7167f81 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.4 / 57fea961ac

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
79 (vs. ulimit of 30000)

modules
C: hbase-zookeeper hbase-server hbase-it . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5490/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

The UT fails consistenctly, even on branch-2.4. Not sure why it does not appear on the flaky dashboard.
@NihalJain FYI, maybe we need your help on backporting the related fix to branch-2.5 and branch-2.4.
Let me merge. Thanks @apurtell  for reviewing!

@NihalJain FYI, maybe we need your help on backporting the related fix to branch-2.5 and branch-2.4.

Sure will have a look and raise backport soon.


Querying Lisp Data-frames?

How much effort would it be for cl-duckdb to query Lisp-Stat data frames?  They are a hashtable of vectors and it looks from the duckdb documentation these might be suitable for direct query scanning, in the same way that Pandas, Julia and R data frames can be scanned.

I was also thinking about something like this. Looking at the Julia bindings this seems to be implemented via a combination of table functions  and replacement scans. These are exposed in the C API, but I didn't get around to looking at them yet: I'll add them to the roadmap.
I'll keep this issue open so related discussion can continue here.

@snunez1: I've cobbled together a proof of concept for table functions based on the C API testcases in DuckDB (see example below, needs the table-functions branch). This is still pretty rough, but I think I can see how table functions could be used to integrate CL data structures more or less directly with DuckDB. I'm not exactly sure how copying data could be avoided altogether, but I'll take a look at what the other bindings are doing.
(in-package :duckdb-api)

(defcstruct my-bind-data-struct
  (size :int64))

(defcallback free :void ((ptr :pointer))
  (foreign-free ptr))

(defcallback my-bind :void ((info duckdb-bind-info))
  (let ((type (duckdb-create-logical-type :duckdb-bigint)))
    (duckdb-bind-add-result-column info "forty_two" type)
    (with-foreign-object (p-type '(:pointer duckdb-logical-type))
      (setf (mem-ref p-type 'duckdb-logical-type) type)
      (duckdb-destroy-logical-type p-type)))
  (let* ((my-bind-data (foreign-alloc '(:struct my-bind-data-struct)))
         (param (duckdb-bind-get-parameter info 0)))
    (with-foreign-slots ((size) my-bind-data (:struct my-bind-data-struct))
      (setf size (duckdb-get-int64 param)))
    (with-foreign-object (p-param '(:pointer duckdb-value))
      (setf (mem-ref p-param 'duckdb-value) param)
      (duckdb-destroy-value p-param))
    (duckdb-bind-set-bind-data info
                               my-bind-data
                               (callback free))))

(defcstruct my-init-data-struct
  (pos :int64))

(defcallback my-init :void ((info duckdb-bind-info))
    (let ((my-init-data (foreign-alloc '(:struct my-init-data-struct))))
      (with-foreign-slots ((pos) my-init-data (:struct my-init-data-struct))
        (setf pos 0))
      (duckdb-init-set-init-data info
                                 my-init-data
                                 (callback free))))

(defcallback my-function :void ((info duckdb-bind-info)
                                (output duckdb-data-chunk))
  (let* ((bind-data (duckdb-function-get-bind-data info))
         (init-data (duckdb-function-get-init-data info))
         (bind-data-size (foreign-slot-value bind-data
                                             '(:struct my-bind-data-struct)
                                             'size))
         (ptr (duckdb-vector-get-data (duckdb-data-chunk-get-vector output
                                                                    0))))
    (with-foreign-slots ((pos) init-data (:struct my-init-data-struct))
      (loop :until (>= pos bind-data-size)
            :for i :from 0
            :do (setf (mem-aref ptr :int64 i)
                      (if (zerop (mod pos 2)) 42 84))
            :do (incf pos))
      (duckdb-data-chunk-set-size output 3))))

(defun register-table-function (connection)
  (let ((function (duckdb-create-table-function))
        (type (duckdb-create-logical-type :duckdb-bigint)))
    (duckdb-table-function-set-name function "my_function")
    (duckdb-table-function-add-parameter function type)
    (with-foreign-object (p-type '(:pointer duckdb-logical-type))
      (setf (mem-ref p-type 'duckdb-logical-type) type)
      (duckdb-destroy-logical-type p-type))
    (duckdb-table-function-set-bind function (callback my-bind))
    (duckdb-table-function-set-init function (callback my-init))
    (duckdb-table-function-set-function function (callback my-function))

    (duckdb-register-table-function connection function)
    (with-foreign-object (p-function '(:pointer duckdb-table-function))
      (setf (mem-ref p-function 'duckdb-table-function) function)
      (duckdb-destroy-table-function p-function))))

;; (ddb:initialize-default-connection)
;; (register-table-function (ddb::handle ddb:*connection*))
;; (ddb:query "SELECT * FROM my_function(3) LIMIT 15" nil)
;; => (("forty_two" . #(42 84 42 42 84 42 42 84 42 42 84 42 42 84 42)))

That was fast progress.  I haven't used CFFI for vectors, but I think that a pointer to them can be passed that will avoid copying.  There seems to be a few different ways.  Tamas Papp described two of them in his blog on FFA.  There's also static-vectors, which is actively maintained.  Maybe cffi:make-shareable-byte-vector is worth looking at too?

@snunez1 thanks for your suggestions, static-vectors look really promising for this. Currently I can't seem to figure out how to avoid copying data via the C API, but I've worked a bit on integrating table functions and the basics seem to be working.
Here's how they can be used. I've stuck with alists for now, but data frames seem very similar in structure.

I took some time to speed up the copying and before even appenders were much faster, now there's a huge difference in favor of table functions:
Results for benchmark MEASURE-INTEGER-APPEND-SUM
-                SAMPLES  TOTAL     MINIMUM   MAXIMUM   MEDIAN    AVERAGE     DEVIATION
REAL-TIME        100      0.459004  0.004     0.01      0.005     0.00459004  7.361473e-4
RUN-TIME         100      0.473135  0.004524  0.010109  0.004642  0.00473135  5.572423e-4
USER-RUN-TIME    100      0.455859  0.00264   0.008146  0.004619  0.00455859  5.4765237e-4
SYSTEM-RUN-TIME  100      0.017198  0         0.002022  0         1.7198e-4   4.369875e-4
PAGE-FAULTS      100      0         0         0         0         0           0.0
GC-RUN-TIME      100      0         0         0         0         0           0.0
BYTES-CONSED<PHONE_NUMBER>8  162560    1076832   162560    186731.69   91499.75
EVAL-CALLS       100      0         0         0         0         0           0.0

Results for benchmark MEASURE-STATIC-TABLE-INTEGER-SUM
-                SAMPLES  TOTAL     MINIMUM  MAXIMUM   MEDIAN   AVERAGE    DEVIATION
REAL-TIME        100      0.079001  0        0.008     0.001    7.9001e-4  8.5200014e-4
RUN-TIME         100      0.090439  7.72e-4  0.008312  8.11e-4  9.0439e-4  7.469862e-4
USER-RUN-TIME    100      0.088499  5.0e-6   0.008286  8.04e-4  8.8499e-4  7.50846e-4
SYSTEM-RUN-TIME  100      0.00165   0        8.05e-4   0        1.65e-5    8.094547e-5
PAGE-FAULTS      100      0         0        0         0        0          0.0
GC-RUN-TIME      100      0         0        0         0        0          0.0
BYTES-CONSED<PHONE_NUMBER>   0        1371728   0        19268.48   136555.05
EVAL-CALLS       100      0         0        0         0        0          0.0

It's still not very robust: https://github.com/ak-coram/cl-duckdb/issues/9#issue-1447050963

@snunez1 I've fixed the issue mentioned above, but I'm afraid we currently can't avoid copying data using the C API. I've opened a DuckDB feature request issue for this: https://github.com/duckdb/duckdb/issues/5339

Do you think duckdb needs another nudge on the zero-copying issue?

I don't think they currently have much interest in this, for cl-duckdb it would also result in some portability challenges. Performance via table functions is already pretty great, so I'm not too concerned about this either right now. Feel free to nugde them though if you think this is important. I'll gladly look into adapting it if it becomes available in the API.

How is it that Julia can do scan's without copying?

How is it that Julia can do scan's without copying?

It doesn't as far as I can tell:
https://github.com/duckdb/duckdb/blob/master/tools/juliapkg/src/data_frame_scan.jl#L79
I think the Python driver does it by calling the C++ API.

Maybe the Julia docs are out of sync?

The DuckDB Julia package also provides support for querying Julia DataFrames. Note that the DataFrames are directly read by DuckDB - they are not inserted or copied into the database itself.

Maybe it's a question of phrasing, the data is not copied "into the database" I guess. I agree that "directly read by DuckDB" makes it seem like it's a zero-copy implementation.

I'm going to close this issue: since there are multiple alternatives for data-frame libraries, I think it's best if the data-frame itself or a third library is used to integrate with cl-duckdb.
See vellum-duckdb as an example of how this could be done: https://github.com/sirherrbatka/vellum-duckdb
Regarding the the zero-copy implementation of table-functions there's not much we can do until DuckDB itself facilitates it in the C API.


Vnda seo

In this Pr I add the advanced Seo end-point in vnda's loader.
Follow the documention: https://developers.vnda.com.br/reference/get_api-v2-seo-data

Are you sure that current endpoints does not receive seo data?
["GET /api/v2/products/search"]
["GET /api/v2/products/:id"]

Yes, I am sure. The folks at vnda were overdue for this configuration for a while.


Read only part 2

It would be nice to have a flag for reading only part 2 of a puzzle.

The underlying aoc-cli does not have a flag for it. It should be simple enough to add this on our side, but I'm wondering if the upstream project wouldn't benefit from this as well.
Anyway, I would accept a PR that adds this on our side as well.

Something I noticed is that once you have completed part 1, the part 2 is not saved automatically to the puzzle file. Doing cargo read <problem_no> after you have completed part 1 makes use of the aoc website/api and reads the puzzle from there but after that it does not save it to the puzzles file again. The puzzle file stays outdated

I opened an issue for tracking this request https://github.com/scarvalhojr/aoc-cli/issues/27.
I'm not entirely sure if the author will get around to reviewing it since it looks like the other PRs from a year ago haven't gotten updated, but I suppose we'll see if that changes. 🤞

I just create a PR for this in the aoc-cli crate https://github.com/scarvalhojr/aoc-cli/pull/28. I'm not sure when it'll get reviewed, but if someone wants this feature now, I can open a PR here. :P

Oh nice, hope this gets merged soon. I would accept a PR that adds it here if there is interest.