(Meta)Data Validation and Document Types

When loading a file I want to validate it against a schema/type so that I know the data/content in my contentbase/"database" is valid

When validation fails what happens?
Error messages should be super helpful
Follow the principle of erroring early

When loading a file I want to allow "extra" metadata by default so that I don't get endless warnings about extra fields that are not defined for document type X.
When accessing a File I want to cast it to a proper typescript type so that I can use it from code with all the benefits of typescript.
BYOT (bring your own types) When working with markdowndb i want to create my own types ... so that when I get an object out it is cast to the right typescript type
Acceptance
TODO
Tasks

[ ] Sketch out a simple example concrete use case (write a short tutorial illustrating this hypothetical feature)
[ ] Research how astro does it with its zod stuff
[ ] sketch out the design before implementing
[ ] ...

Design

Should the user define the scheme using Zod, or should we build a more intuitive way that doesn't require knowledge of external libraries?

I believe we should make the use of Zod optional, as it may be overkill for users. Most users likely want to validate if a specific front matter field is provided, rather than engaging in complex validation.
The best approach would be to allow users to:

Make a field required or optional.
Specify whether it's a string or a number.
Provide their own validation logic (giving users flexibility without imposing unnecessary complexity).
Choose whether to use Zod for validation.

Example:
If a user wants to validate that a field named dates with two or more dates is provided, the schema could be defined as follows:
dates: {
    type: string,
    required: true,
    validate: (fileObject) => {
        if (/* field 'dates' is incorrect */) {
            return {
                status: false, // Error
                message: ""     // Error message
            };
        } else {
            return {
                status: true,  // Correct
            };
        }
    }
}

Should all the file schemes be defined in the config file?
Yes, it's cleaner, and we should allow the users to import schemes from other js files for organization.

Determining Files Matching a Scheme:

Option 1: Continue with type Property

If no pattern field is provided, default to using the type property (e.g., type: "post"). This caters to simplicity for basic users and use cases.

Option 2: Introduce a pattern Property

If a pattern field is provided, utilize it to match files based on the specified pattern (e.g., pattern: "post/**"). This offers flexibility for handling diverse and complex use cases.

Consideration:
The goal is to strike a balance between providing essential functionality and accommodating complex use cases. The defaulting to type for simplicity while allowing the use of a custom pattern enables users to tailor the file matching process to their specific needs.

Adding Computed Fields:

To incorporate computed fields into the scheme, a compute or value field can be introduced, accepting a function. This allows users to perform computations or set values dynamically.
Example:
Suppose you have a scenario where the dates field needs to be computed based on some dynamic logic. You can define it as follows:
dates: {
    type: string,
    required: true,
    compute: (fileObject, ast) => {
        // Perform dynamic computation based on fileObject or ast
        // ...

        // Return the updated fileObject
        return fileObject;
    },
    validate: (fileObject) => {
        // Validation logic for the 'dates' field
        // ...
    }
}

In this example, the compute function takes the fileObject (representing the file's metadata) and ast (abstract syntax tree) as parameters. Users can apply custom logic within the compute function to dynamically calculate or set the value of the dates field.
This approach enhances flexibility by allowing users to include computed fields seamlessly within the overall scheme, contributing to a more versatile and adaptable file structure.

Also Maybe we can build npm packages with a pre-defined library of schemes or provide a comprehensive list of examples, as I consider schemes to be something not that simple for all users.
I think a few examples that cover what the user might want to do is enough.

Notes
Architecture
I think we use zod here and typescript types. (cf astro approach)
How does zod work?

Defining Schemas:

You create a schema using the various methods provided by Zod, such as z.string(), z.number(), z.object(), etc.
Example:import { z } from "zod";

const userSchema = z.object({
  username: z.string(),
  age: z.number().min(18),
});

Validation:

You can then use the parse method to validate and parse data according to the defined schema.

Example:
const validUserData = userSchema.parse({
  username: "john_doe",
  age: 25,
});

If the provided data doesn't match the schema, a ZodError is thrown with details about the validation errors.

Custom Error Messages:

You can customize error messages to provide meaningful feedback to users.

const userSchema = z.object({
  username: z.string().min(3, { message: "Username must be at least 3 characters" }),
  age: z.number().min(18, { message: "Age must be at least 18" }),
});

@rufuspollock I've advised @mohamedsalem401 to first sketch this out before writing any code

@rufuspollock
Let's explore a straightforward example involving a content management system responsible for storing blog posts. Each blog post adheres to a specific structure, encompassing properties such as title, content, tags, and author. Our focus is on devising the file loading process, employing the previously delineated concepts.
// blogPostSchema.json
{
  "pattern": "blog/*.md",
  "properties": {
    "title": { "type": "string", "required": true },
    "date": { "type": "string" },
    "author": { "type": "string", "default": "Anonymous" },
    "tags": {
      "type": "array",
      "items": { "type": "string" },
      "minItems": 1,
      "uniqueItems": true
    }
  },
  "additionalProperties": false // allow extra metadata (true by default)
}

In the event that a user attempts to execute the indexFolder function and a file fails to adhere to the specified schema, an error message like the following would be displayed:
Error: Title field is required on type Post at blog/post1.md

When it comes to comparing schemas, TypeScript proves to be a valuable tool. By converting the schema into a TypeScript interface, we can facilitate straightforward comparisons. Consider the following examples:
Schema:
{
  "type": "object",
  "properties": {
    "title": { "type": "string", "required": true },
    "content": { "type": "string" },
    "author": { "type": "string" }
  }
}

Equivalent TypeScript Interface:
interface BlogPost {
  title: string;
  content?: string;
  author?: string;
}

This approach allows us to define TypeScript interfaces mirroring the JSON schema, enabling us to harness the TypeScript type system for identifying potential type errors during development.

@mohamedsalem401 what does zod do in astro build?

@mohamedsalem401 i was more thinking about the use of zod in the astro content collections feature which is similar to what we have in markdowndb https://docs.astro.build/en/guides/content-collections/

Configuring Content Collections in src/content/config.ts:

In the src/content/config.ts file, the user configures content collections for an Astro project.
The configuration involves specifying the type of collection and defining a schema for frontmatter validation.

Using zod for Schema Validation:

Astro utilizes zod directly for schema validation.
Users need to learn how to use zod to define and validate the structure of frontmatter in their content collections.

Searching for Astro Configuration Files:

There's a function that searches for Astro configuration files (astro.config.js|cjs|mjs|ts) in different formats.
This function operates within a specified root directory.

Loading Astro Configuration with Vite:

Another function loads the Astro configuration using Vite, a frontend build tool.

Defining Collections in src/content/config.ts:

Collections are defined in the src/content/config.ts file using the defineCollection function.
All defined collections are exported as part of the collections object.

Querying Documents of Type "blog":

To retrieve documents of type "blog," there's an example usage of the getCollection function.
The function is invoked with the argument 'blog' to fetch all entries from the "blog" collection.
The result, stored in allBlogPosts, contains the content entries from the specified collection.

In summary, the user configures content collections in src/content/config.ts, defines collection types and schemas using zod, and utilizes Astro functions like getCollection to query and retrieve content entries based on the defined collections.

Some of the questions that we might answer:

Should the user define the scheme using Zod? Or should we build a more intuitive way that doesn't require the knowledge of external libraries?

Should all the file schemes be defined in the config file?

Astro knows the files that match a file scheme by the type property. For example, a scheme with type: "post" will make all files in content/post/** become post files. Should we do this? Or use a pattern property, e.g., a scheme with pattern: "post/**" will make all files in content/post/** become post files.

How to add computed fields in all of this? Maybe we can do something like:

function getContentLength(data) {
  return data.content.length;
}

scheme: {
    content: z.string(),
    length: {
      scheme: z.string().optional(),
      value: getContentLength
    }
}

or
function getContentLength(data) {
  return data.content.length;
}

scheme: {
    content: z.string(),
    length: z.string().optional().transform(getContentLength) // pass the function directly to `zod`'s `transform` method
}

Also Maybe we can build npm packages with a pre-defined library of schemes or provide a comprehensive list of examples, as I consider schemes to be something not that simple for all users.

@rufuspollock
I have added answers for how I would address the document types design in the Design section and also explained how Zod works in the notes section.

@mohamedsalem401 now we have a basic sketch let's start creating a really simple example where we use zod to set types and then have validate and reporting on build of markdown in some useful way. This could start as a PR an example in the examples folder. I've added stuff to acceptance for you.


Initialize Typescript upgrade

New Dependencies

dotenv

New Dev Dependencies

typescript
@types/body-parser
@types/cors
@types/express
@types/lodash
@types/memory-cache

New Scripts

build to compile typescript code to javascript dist directory
watch to run typescript build in watch mode
local to run locally without pm2

Removed

Husky Hooks (Require discussion, if we want to keep it with Typescript)

Changes

.js files renamed to .ts and type any applied  everywhere for now.
API code moved to src directory
tsconfig.json added

Nest Steps

Require feedback and the end to end testing

I am not seeing any logic changes so LGTM.
I can't test as both npm run test and npm run dev do not work. They throw errors such as:
(Use `node --trace-warnings ...` to show where the warning was created)
(node:66469) Warning: Accessing non-existent property 'cd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'chmod' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'cp' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'dirs' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'pushd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'popd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'echo' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'tempdir' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'pwd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'exec' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'ls' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'find' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'grep' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'head' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'ln' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'mkdir' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'rm' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'mv' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'sed' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'set' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'sort' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'tail' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'test' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'to' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'toEnd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'touch' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'uniq' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'which' of module exports inside circular dependency```

I am not seeing any logic changes so LGTM.
I can't test as both npm run test and npm run dev do not work. They throw errors such as:
(Use `node --trace-warnings ...` to show where the warning was created)
(node:66469) Warning: Accessing non-existent property 'cd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'chmod' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'cp' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'dirs' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'pushd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'popd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'echo' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'tempdir' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'pwd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'exec' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'ls' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'find' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'grep' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'head' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'ln' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'mkdir' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'rm' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'mv' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'sed' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'set' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'sort' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'tail' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'test' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'to' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'toEnd' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'touch' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'uniq' of module exports inside circular dependency
(node:66469) Warning: Accessing non-existent property 'which' of module exports inside circular dependency```

I do not know how process-dev.json file looks like so haven't tested it with pm2 maybe we will have to modify it to take code from dist directory or something else.
test may need some rework as we have made the major structural changes. It shall be the next task after we get this merged.
I would also like to propose it as v3 release.


Not working for the whisperer, all works but not her name

Not working for the whisperer, all works but not her name

Try adding Odd figure as well.

It worked by adding both Odd Figure and The Whisperer, but not if i added just one! thank you! i spent like an hour trying to fix it lololol


changed yt-dl.org to youtube-dl.org (binary downloads were failing)

youtube-dl.org is the official domain for youtube-dl.
More importantly the TLS certificate of yt-dl.org isn't valid, causing downloads of the binary to fail

Hey, @ArthurHeitmann thanks for this.


Monorepo transition feedback

This issue is for collecting feedback about the monorepo setup. This feedback should be turned into documentation for contributors on the project.

It looks like the build environment is failing because it can't find a certain package that is in the monorepo, did lerna/yarn not initialize it correctly?

Most likely this is due to not running yarn build when first starting up the project. yarn build is important to initialize generated packages from elements, and it creates the entrypoint for a number of packages.


Swiss database contains calculation errors

The current supply_system database includes the following values (the units are not shown in the database, so they are my assumption):
T1 - swiss consumer energy mix: 2.63 kWh PEN/kWh and 0.0413 kg CO2/kWh
T2 - PVT panel - monocrystalline roof top: 0.278 kWh PEN/kWh and 0.0202 kg CO2/kWh
T3 - PVT- monocrystalline roof top: 0.222 kWh PEN/kWh and 0.0108 kg CO2/kWh
T4 - swiss consumer energy mix 70% + 30% PV: 0.1535 kWh PEN/kWh and 0.0002 kg CO2/kWh
T5 - swiss consumer energy mix  70% + 30% PVT: 0.1226 kWh PEN/kWh and 0.0001 kg CO2/kWh
I see a couple of problems with these. First of all: T2 and T3 are basically defined as the same thing. T2 should be PV and T3 PVT?
Secondly and more importantly: How can a combination of the consumer energy mix and solar panels have a lower primary energy and CO2 emissions per kWh than either the consumer energy mix or the solar panels alone? Well, T4 and T5 are calculated as follows:
PEN_T5 = 0.7 * PEN_T1 * 0.3 * PEN_T3 and CO2_T5 = 0.7 * CO2_T1 * 0.3 * CO2_T3
i.e., instead of adding the two, they are multiplied, so the results are lower than they should be.
I am now fixing both of these errors. However, @JIMENOFONSECA: could you provide the source for these energy mixes used? It would be good in any case to include the source in the manual or somewhere else. This way we can try to be consistent with this source when defining other systems and energy mixes (e.g., Belgium).

I just corrected the mistakes in the Excel file. The only thing missing is the source of the factors used.

Hi, the source are the ökobilanzdaten analysis of the kbob for buildings
https://www.kbob.admin.ch/dam/kbob/de/dokumente/Publikationen/Nachhaltiges Bauen/Empfehlung_Oekobilanzdaten_Baubereich_2009_1_2014.pdf.download.pdf/Empfehlung Ökobilanzdaten im Baubereich 2009/1_2014.pdf
Best, Jimeno
On 23 Sep 2016, at 22:06, martin-mosteiro <notifications<EMAIL_ADDRESS>wrote:
The current supply_system database includes the following values (the units are not shown in the database, so they are my assumption):
T1 - swiss consumer energy mix: 2.63 kWh PEN/kWh and 0.0413 kg CO2/kWh
T2 - PVT panel - monocrystalline roof top: 0.278 kWh PEN/kWh and 0.0202 kg CO2/kWh
T3 - PVT- monocrystalline roof top: 0.222 kWh PEN/kWh and 0.0108 kg CO2/kWh
T4 - swiss consumer energy mix 70% + 30% PV: 0.1535 kWh PEN/kWh and 0.0002 kg CO2/kWh
T5 - swiss consumer energy mix 70% + 30% PVT: 0.1226 kWh PEN/kWh and 0.0001 kg CO2/kWh
I see a couple of problems with these. First of all: T2 and T3 are basically defined as the same thing. T2 should be PV and T3 PVT?
Secondly and more importantly: How can a combination of the consumer energy mix and solar panels have a lower primary energy and CO2 emissions per kWh than either the consumer energy mix or the solar panels alone? Well, T4 and T5 are calculated as follows:
PEN_T5 = 0.7 * PEN_T1 * 0.3 * PEN_T3 and CO2_T5 = 0.7 * CO2_T1 * 0.3 * CO2_T3
i.e., instead of adding the two, they are multiplied, so the results are lower than they should be.
I am now fixing both of these errors. However, @JIMENOFONSECAhttps://github.com/JIMENOFONSECA: could you provide the source for these energy mixes used? It would be good in any case to include the source in the manual or somewhere else. This way we can try to be consistent with this source when defining other systems and energy mixes (e.g., Belgium).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/architecture-building-systems/CEAforArcGIS/issues/345, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIjrgsnpufbyaCgovWBSi6iW6D9EO4mUks5qs9zZgaJpZM4KFBEB.

btw, I am afraid something is happening with the databases. This error was corrected about twice already last year... did you modified the master?

I had modified the master, yes. My apologies for not creating a branch, I should have done that.

@martin-mosteiro no problem, next time..


Have plugins use the same $PATH as current user?

Scripts are currently run using a different $PATH, and so all terminal commands are not available to me unless I use export PATH='/my/paths:/my/additional/paths'
I'm using node to produce my script outputs, and node was not available to me until I changed the path or specified the node executables full path.

Whoops. This is a duplicate of https://github.com/matryer/bitbar/issues/36


distance() on 3D geometries silently projects to 2D

Hi there, and thanks for making such an incredibly useful product!!!
I wanted to let you know about some math that appears to be broken in this library.
Issue:
Despite allowing 3D geometries [e.g. Point(0,0,0) and LineString(((0,0,1), (1,1,0))], when I try to do any Cartesian math with them, the result seems to first (silently) project back down to 2D, by throwing away the Z coordinate, before returning the result.
Example:

Point(0,0,0).distance(LineString(((0,0,1), (1,1,0))))

Expected Behavior:
In 3D Cartesian math, this should should return "0.81649..." or sqrt(2/3).
However, in shapely it returns "0" (even though you can see these two geometries do not intersect).
Desired Behavior:
Ideally, the library would do the actual calculations in 3-space.
If it can't do that, then it should at least raise a warning anytime it does an automatic discarding of data (e.g. removing the Z coord), and then proceed with the 2D calculation.
pygeos has the same undesirable behavior, so I don't expect it will be resolved automatically during the integration.
Thanks, and apologies if you've already addressed this in another issue!

Oops!
Of course, immediately after posting, I found this https://github.com/shapely/shapely/issues/613
I do, of course, understand that this is intended behavior (as described in the shapely docs), but nonetheless, it seems like an extremely useful extension to existing functionality, and is implicitly implied by the ability to do math on 3D objects.
Perhaps you might just consider this note as an additional request for this feature (3D math) to be supported.

even though you can see these two geometries do not intersect

Note that also for predicates like intersects, shapely (GEOS) will ignore the 3rd dimension, so while I know those don't intersect in 3D, shapely will actually say they do:
In [6]: Point(0,0,0).intersects(LineString(((0,0,1), (1,1,0))))
Out[6]: True

While distance might be relatively simple in 3D, other operations like those predicates or overlays are not at all, so I suppose that is the main reason this is not supported. An actual feature request would also need to be directed to https://github.com/libgeos/geos

Thanks for the additional information, @jorisvandenbossche!
Just curious if you know offhand of any Python libraries that are nearly as powerful as shapely, but are designed to handle 3D math properly?
I always think of shapely as the leader in the field, but it seems like I'll need to patch over this hole in functionality by bringing in another library.

scikit-geometry (https://scikit-geometry.github.io/scikit-geometry/index.html) might handle this, as CGAL bindings (https://www.cgal.org/). But I have no experience with that, so can't say for sure.


Error running build.cmd deploy

Hello,
running build.cmd deploy I get some errors.
I'm not able to create nuget packages because not all builds are completed before creating nuget packages.
I have attached my console output.
I'm using msbuild v.14.0. I suppose that something in the order of compilation was wrong.
Best regards,
Filippo.
BuildCmdLog.pdf

Hi Filippo,
I've fixed the build issues in https://github.com/gimelfarb/ProductionStackTrace/pull/13.
I hope it helps you.


ILMerge.MSBuild.Task.targets incompatible with VS2017

It seems that we need a minor modification to the file in order for VS2017 to recognize it properly.
When loading the target file, MSBuild shipped with VS2017 complains that the %(CopyLocal) should be qualified as %(ReferencePath.CopyLocal).
There are two places in the file to which this qualification applies.
I would be glad if you could update the nuget package with this fix applied, though the fix is still possible at the project level after installing the package.
I thought it was overkill to fork the entire repository and issue a pull request just for this fix.
Thank you in advance.

Could you please give me more details on when and where you see this error? I've been running this on VS2017 for a while (actually I use VS2017 since it came out) and I don't have any issues. I will be glad to fix (ASAP) it when I understand where and why exactly it is happening.


Maven dependcy not found

Hi just included it to my pom.xml but Intellij did not found it :( Is there something I looked over?
        <dependency>
            <groupId>net.ricecode</groupId>
            <artifactId>string-similarity</artifactId>
            <version>1.0.0</version>
        </dependency>

Did you install the project in your local repository? Check Installation part from README.md. Artifacts of the project aren't published in Maven Central.

That is correct. This project does not publish to Maven Central because the project is not fully compliant with the publishing requirements. I normally publish this project on a private server, so the project that need.  However, if there is enough demand, I would not object to polish up the project to meet these requirements.

Ahhh now I get it. Sorry for my question im mostly new to maven and java. Thank you for the quick reply and btw: Nice project! I use it for this https://github.com/frankred/jSona  and working to switch to maven. Good times.

Hi there,
@rrice i would like to help publishing this library on maven central.
I already published some libs on central and could help to do this with that one.
Are you interested in doing this? I want to use this library but without availability in the central i can't, due to company's internal limitations.

@renatorroliveira Absolutely. I am starting to some work now to meet the requirements outlined here.  I could use some help with getting the pom files set up correctly.  So, I will accept pull request that helps achieve this goal. Thanks!

Great!
I will setup the POM for this purpose.
@rrice While i do this, i suggest you open a ticket on Sonatype JIRA to create the projects/repositories for your groupId:
http://central.sonatype.org/pages/ossrh-guide.html#create-a-ticket-with-sonatype

The JIRA ticket is filed under OSSRH-19996

Great, it is already created. I will proceed with the POM.

@rrice Can you contact me via e-mail? You will need to prepare your environment to release versions of the library to the central, and that include generating GPG key pairs and others confidential stuff.

@renatorroliveira I have already created and updated my GPG key pairs. I will contact you soon to discuss any publishing requirements that I have missed.  Thanks!

Hi @rrice,
Very good! So i will post here some guidelines regarding my pull request (#6) to ease the publishing process.
In my published libraries, i configure the GPG and Sonatype repositories in the Maven's settings.xml (~/.m2/settings.xml). Below is an example configuration where you have to put the information of your  GPG key and Sonatype OSS account:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository/>
  <interactiveMode/>
  <usePluginRegistry/>
  <offline/>
  <pluginGroups/>
  <servers>
    <server>
      <id>ossrh</id>
      <username>YOUR_SONATYPE_USERNAME</username>
      <password>YOUR_SONATYPE_PASSWORD</password>
    </server>
  </servers>
  <mirrors/>
  <proxies/>
  <profiles>
    <profile>
      <id>ossrh</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <gpg.executable>gpg2</gpg.executable>
	<gpg.keyname>E994461B</gpg.keyname>
        <gpg.passphrase>YOUR_KEY_PASSPHRASE</gpg.passphrase>
      </properties>
    </profile>
  </profiles>
  <activeProfiles/>
</settings>

You can get the gpg.keyname property listing your keys with the gpg program.
With that configured, i usually use the Maven Release Plugin to deploy the artifacts to the repositories. In the POM i configured to sign the artifacts when you activate the release profile, so to deploy the artifacts you can execute:
mvn release:prepare -P release
mvn release:perform -P release

If the process goes well, then a staging repository will be created at Sonatype OSS (http://oss.sonatype.org/). You have to close the staging repository and if it succeeds, you can release it. After releasing, in about 15 minutes the artifacts will be available at Maven Central Repository (http://central.maven.org/maven2/net/ricecode/).
If you have any problems just contact me =)

Looks like the component is published to central:
http://repo1.maven.org/maven2/net/ricecode/string-similarity/1.0.0/
Thank you @renatorroliveira for your assistance.

@rrice Awesome news!
Now we can just add the dependency in the POM =D

AWESOME! THX!


Live-collaboration is broken on excalidraw.com

Hi! Thanks for this great software. Sadly, the live-collaboration feature seems broken at the moment due to CORS issues. When opening the developer console, I get the following errors:
Access to XMLHttpRequest at 'https://portal.excalidraw.com/socket.io/?EIO=3&transport=polling&t=XXX&sid=' from origin 'https://excalidraw.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

trycatch.ts:281 POST https://portal.excalidraw.com/socket.io/?EIO=3&transport=polling&t=XXX&sid=XXX net::ERR_FAILED

I have observed these errors with both Firefox and Chromium.

Same issue for me, Chrome.

Same issue for me, Chrome.

I labeled it as bug, but I actually can't reproduce it. I was able to do so on Firefox, but it seems I was on an old Excalidraw version.
I'm not seeing any CORS errors between portal.excalidraw.com and excalidraw.com domains :(

I labeled it as bug, but I actually can't reproduce it. I was able to do so on Firefox, but it seems I was on an old Excalidraw version.
I'm not seeing any CORS errors between portal.excalidraw.com and excalidraw.com domains :(

I labeled it as bug, but I actually can't reproduce it. I was able to do so on Firefox, but it seems I was on an old Excalidraw version.
I'm not seeing any CORS errors between portal.excalidraw.com and excalidraw.com domains :(

@dwelle  How about trying this on the latest Excalidraw version?

I labeled it as bug, but I actually can't reproduce it. I was able to do so on Firefox, but it seems I was on an old Excalidraw version.
I'm not seeing any CORS errors between portal.excalidraw.com and excalidraw.com domains :(

@dwelle  How about trying this on the latest Excalidraw version?

I did. No errors. We've tried collaborating among several team members and had no issues either.
I've now noticed the error message you supplied mentions it's related to credentials. Reading up on it, I wonder why it's not reproducing for everyone, as well. I'll see what we can do.

I did. No errors. We've tried collaborating among several team members and had no issues either.
I've now noticed the error message you supplied mentions it's related to credentials. Reading up on it, I wonder why it's not reproducing for everyone, as well. I'll see what we can do.

Ok, I think it's because your browser must be stripping request headers. Do you have a browser extension that might be doing that? Some ad blocker or other privacy related extension.

Ok, I think it's because your browser must be stripping request headers. Do you have a browser extension that might be doing that? Some ad blocker or other privacy related extension.

Ok, now I'm really confused. We've juste tried it again and it works!

Ok, now I'm really confused. We've juste tried it again and it works!

Good. Nevertheless, we'll make some changes that should fix that error in the future.

Good. Nevertheless, we'll make some changes that should fix that error in the future.


CUDA out of memory

RuntimeError: CUDA out of memory. Tried to allocate 1.21 GiB (GPU 0; 3.95 GiB total capacity; 1.27 GiB already allocated; 889.19 MiB free; 1.27 GiB reserved in total by PyTorch)

@oscarknagg @RajeshPandey057 @LIAOJUNJIE-OBJ , When I run the experiments.proto_nets on the dataset miniImageNet, it reports RuntimeError: CUDA out of memory. Though I have modified the parameters episodes_per_epoch and num_workers to make them smaller, this error still occurs. How can I resolve this problem? Thank you very much.


TargetHeaterCoolerStateCharacteristic: Error parsing state Error: Unsupported API value: 4

LG Air Conditioner Model
S12EW
Describe the bug
from homebridge Log, msg from subject
Symptom check

Does your unit appear in the Home app? Yes
Is the current on/off state, temperature, and mode being reported? Yes
Does your unit respond to the controls? Yes/No

Relevant logs
If you feel there are relevant logs, please include them here. Otherwise you can remove this.

Is your AC unit on the “energy saver” mode?

Also if you open the accessory details on the Homekit app, what is the model that is displayed?

hi stuart, do you want to do some online debugging on whatsapp? if yes +40728200000

no energy saving mode, write now it’s on heating
model RAC_056905_WW

On 5 Jun 2020, at 08:31, Stuart Olivera<EMAIL_ADDRESS>wrote:
Also if you open the accessory details on the Homekit app, what is the model that is displayed?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/sman591/homebridge-lg-thinq-ac/issues/22#issuecomment-639268497, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDSXDK73HR7RTYAJIGBR4LRVB7LJANCNFSM4NTHMQPA.

hi stuart,

no energy saving mode, write now it’s on heating
model RAC_056905_WW

The same issue appear on LG B09TS

after last update this error wasn't seen instead appeared "SwingModeCharacteristic: Error parsing state Error: Unsupported API value: 6"

@dimabobrov @cretum2108 Does the "cooling" mode work properly? Does setting to "auto" change to the fan-only mode?

@dimabobrov For your LG B09TS model, what model is displayed on Homekit's accessory details?

Hi Stuart, unfortunately my air had an issue and i had changed it to this model https://altex.ro/aer-conditionat-cu-purificator-lg-ap12rt-12000-btu-a-a-wi-fi-alb/cpd/AERAP12RTNSJ/

@dimabobrov @cretum2108 Does the "cooling" mode work properly? Does setting to "auto" change to the fan-only mode?

Yes, the "cooling" mode works properly
Yes, setting to "auto" changes the mode to "fan-only"

@dimabobrov For your LG B09TS model, what model is displayed on Homekit's accessory details?

The model is "RAC_056905_WW"

some more debug info
[LgThinqAirConditioner] TargetHeaterCoolerStateCharacteristic: Warning: Your model may support a "drying" or "dehumidification" mode. This is NOT natively supported by Homekit, and using it may show errors in Homebridge or cause temporary instability.
[LgThinqAirConditioner] Error renewing monitor Error: Request failed with status code 400

Here is the mapping of the modes to TargetHeaterCoolerStateCharacteristic
Cooling: 0
Drying: 1
Fan-Only: 2
Heating: 4
Auto: 6


re-export stake::state::StakeAuthorize enum

used in https://github.com/igneous-labs/unstake/pull/11

decided not to re-export from the module


Expand Calico's failsafe ports, for a k8s hosted install

When Calico is installed using manifests into a k8s cluster, using its own etcd database, it makes sense to configure failsafe port lists so that the base system will continue working if host endpoints are defined without any additional policy.  (Notwithstanding the excellent advice at http://docs.projectcalico.org/master/getting-started/bare-metal/bare-metal#creating-basic-connectivity-and-calico-policy.)
Specifically:

In this kind of install, Calico's etcd will be using a different port number from the etcd standard ones.  That port number should be allowed.
In this kind of install, Calico's etcd will run as a k8s service, so its port number needs to be allowed both inbound and outbound.
The BGP port number should be allowed as a failsafe, both inbound and outbound.

Expected Behavior
If a host endpoint is defined, it should still work to run calicoctl from a cluster node.
Current Behavior
If a host endpoint is defined, calicoctl times out after failing to connect to the calico-etcd service.

In general there are 2 things we can do here:

Expand Felix's hardcoded defaults for the failsafe ports.  That would obviously then affect all deployments.

Add environment settings to particular manifests, so as to add failsafe ports for those particular deployments.

Felix's current hardcoded defaults are:
	FailsafeInboundHostPorts  []ProtoPort `config:"port-list;tcp:22,udp:68;die-on-fail"`
	FailsafeOutboundHostPorts []ProtoPort `config:"port-list;tcp:2379,tcp:2380,tcp:4001,tcp:7001,udp:53,udp:67;die-on-fail"`

ISTM there is a good case for adding 179 (bgp) to these defaults, in both directions.
For the etcd port number in a k8s install, I suspect using manifest settings is better.  The current defaults already make sense for a deployment in which the etcd nodes are distinct from the workload hosts.
CC @fasaxc @song-jiang  WDYT?

Yes, that makes sense to me.  Needs to be a manifest setting, it's going to be different for each type of deployment.

Looks to me like this has been addressed, so closing this. Reopen if I missed something!


undeclared modules in pallets

Hi,
Have anyone tested clicking on "use this template" and then running the cargo build --package minimal-template-node --release ?
Below is the screenshot of the user experience.
.
First command, and it fails.
May i suggestion a CICD checker service ?

Hello,
The repo does have CICD checks:

I just checked and it works for me locally too.
What is your rust compiler version? Perhaps there is something there.

Hi,
My rust compiler version:
➜ rustc --version
rustc 1.78.0 (9b00956e5 2024-04-29)
my rust active tools:
➜ rustup show
Default host: aarch64-apple-darwin
rustup home:  /Users/zhenhaowu/.rustup
installed toolchains
stable-aarch64-apple-darwin (default)
nightly-2022-09-14-aarch64-apple-darwin
nightly-2022-11-14-aarch64-apple-darwin
nightly-2022-12-10-aarch64-apple-darwin
nightly-2023-05-22-aarch64-apple-darwin
nightly-2024-02-16-aarch64-apple-darwin
nightly-2024-04-22-aarch64-apple-darwin
nightly-aarch64-apple-darwin
1.70-aarch64-apple-darwin
1.74-aarch64-apple-darwin
1.75-aarch64-apple-darwin
1.76-aarch64-apple-darwin
installed targets for active toolchain
aarch64-apple-darwin
wasm32-unknown-unknown
active toolchain
stable-aarch64-apple-darwin (default)
rustc 1.78.0 (9b00956e5 2024-04-29)
I had a friend try the procedure, "use this template" followed by the build command, we both have the same error.

@DarkArtistry The issue has been corrected. There actually was a difference between the cargo build --package minimal-template-node --release command from the readme and the cargo build --release command.
Sorry about your bad experience.


Fix RoleDescriptor field annotations for Serverless

I included these changes for SecurityPutRoleRequest but they also need to be part of the role descriptor model since that is referenced by e.g. the API key APIs.

Note that you don't need to click on the "Update branch" button whenever the branch is out-of-date, especially as today is a busy day regarding merges.

Note: RoleDescriptor is missing remote_indices and remote_cluster field definitions -- I will add these in a follow up PR.

Backport:

https://github.com/elastic/elasticsearch-specification/pull/2702


update docs for TabbedInterface

This micro-PR just updates the docs for the TabbedInterface to be consistent with the other places that it's documented (by including a heading kwarg)
Correct
https://docs.wagtail.io/en/v2.5.1/reference/contrib/settings.html?highlight=tabbedinterface
vs
Borderline Incorrect
https://docs.wagtail.io/en/v2.5.1/reference/contrib/modeladmin/create_edit_delete_views.html?highlight=tabbedinterface
It's worth noting that without a heading field the tabbed interface doesn't work properly because the javascript on the page uses an identifier of #tab-{{ heading }} -- this means if all of the tabs have the same default '' for the heading -- they won't switch properly.
This means it could be worth making heading a required field for the tabbed interface! But that would be a slightly bigger change (and I'd be willing to take a look tomorrow)
Thanks!

Thanks @zconnelly13! Merged in a19571db7acc40e7002e4e2059f1e3b4479857b9.


ISPN-9224 Reduce ExceptionEvictionTest combinations

Comment out OFFHEAP/BINARY cluster mode options

https://issues.jboss.org/browse/ISPN-9224

@wburns, I've looked a little more into this and I think it would be better to keep all the configurations for the base test if we split off the expiration, state transfer, and overflow on scale down test methods into separate classes that only deal with OBJECT storage (or BINARY, or OFF_HEAP if you think it would be more interesting).
There's also a small inefficiency in clearContent(), which invokes cache.clear() on all the caches but without CACHE_MODE_LOCAL. Just setting the interceptor field saved me almost 1 minute (w/ trace logging enabled). Weird, because it's 1 call instead of 3, calling just cache(0).clear() got me almost exactly in the middle, with 30s saved vs. master.
So even without the OffHeapMemory logging, the extra data container iterations seem quite expensive. I wonder if we could make it faster, maybe we could implement forEach() in OffHeapDataContainer to avoid the stream overhead?
We could also skip the data container iteration in ClearCommand and AbstractClusterDependentLogic if there are no remove listeners, but I'm not sure how I'd like a remove-specific method in CacheNotifier, I'd like a more generic solution there for interceptors to also skip loading the previous value when possible. Maybe CacheImpl could set the SKIP_CACHE_LISTENERS flag on the command when there are no listeners registered at command start? Or maybe we'll even change clear() to not notify listeners at all...

Forgot another option, maybe we could use a smaller addressCount in tests and/or when we know the size limit?

Reducing the address count to say 1 << 7 cut the time to a third. So I can just change that.

Updated.

Merged


No such file or directory exception

Hi,
I can't run cage on Macos, zsh, Python 3.5. My steps was like on readme file except Python is in virtual environment:

pip install pycage
cd /tmp/application && touch test.py
cage app:create something

Traceback:
Traceback (most recent call last):
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
    chunked=chunked)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 363, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/docker/transport/unixconn.py", line 33, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 640, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 261, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
    chunked=chunked)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 363, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/docker/transport/unixconn.py", line 33, in connect
    sock.connect(self.unix_socket)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/psv/.env/qa-automation-web/bin/cage", line 11, in <module>
    sys.exit(main())
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/cage/__init__.py", line 25, in main
    handle_container_command(command_list[1], args)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/cage/__init__.py", line 44, in handle_container_command
    res = container_handler.create_image(python_version)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/cage/container/handler.py", line 75, in create_image
    response = self.__client.build(path=self.__app_path, tag=self.__image_name, rm=True)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/docker/api/build.py", line 105, in build
    timeout=timeout,
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/docker/utils/decorators.py", line 47, in inner
    return f(self, *args, **kwargs)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/docker/client.py", line 135, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/sessions.py", line 522, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/Users/psv/.env/qa-automation-web/lib/python3.5/site-packages/requests/adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Hi,
Thanks for reporting this!
This exception is thrown if Docker is not running.
I'll leave this issue open until I put in a better message.

Starting Docker before running any cage commands will fix this for now.

Thank you for reply! What do you mean "Docker is not running"?
Docker daemon is running on my machine.
psv@psv:/tmp|⇒  eval $(docker-machine env)
psv@psv:/tmp|⇒  docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

It seems you have the Docker Toolbox installed on your system.
Cage works with Docker for Mac as it uses the exposed socket to connect to the daemon.
I recommend you use Docker for Mac instead of the Toolbox as it also comes with some cool updates like not using VirtualBox for the containers.
See this page for more info:
https://docs.docker.com/docker-for-mac/docker-toolbox/

Closing this issue and added an Enhancement to specify the supported Docker versions in the README file: #2


Decomposing a 3D matrix

First of all, great job with this library - the code is well-documented and clear. I was up late last night looking for a good library for manipulating CSS transforms, and was very happy to find this one!
I was wondering if you would be open to the following (small) features? I can definitely help with some of them, let me know if you are willing to accept PRs on any of these.

A method for converting a matrix array to a CSS string, e.g., Rematrix.toString([...]) or more explicitly, Rematrix.toCSSValue (or any other appropriate name)

would accept an array and return a string, e.g., "matrix3d(0, 0, ...)" etc.

Rematrix.rotate(...) that defaults to Rematrix.rotateZ(...) (same as the CSS rotate(a) spec)
Something that allows you to specify the unit for rotation, such as rad or turn

Rematrix.rotate(0.5, 'rad') perhaps?

Decomposition - probably a bigger feature ask, but a way to decompose a matrix and grab specific values such as translateX, skewY, rotateZ, scaleY, etc. from it?
Typescript definition file - I can probably PR this quickly.

Thanks so much for this library!

Thanks for the kind words @davidkpiano, I appreciate it!

"Decomposition - probably a bigger feature ask, but a way to decompose a matrix and grab specific values such as translateX, skewY, rotateZ, scaleY, etc. from it?"

Now this is 🔥, and strikes me as a very powerful addition to the API. Some cursory research has revealed it should be possible using inverse trigonometric functions.
I don’t mind sharing my thoughts on your other ideas, but I found decompose() so compelling it eclipsed the rest. I think it adds the most value for developers. If you’re up for it, I think it would make an outstanding contribution.

Sure, and for reference the W3C has a decomposing algorithm: https://www.w3.org/TR/css-transforms-1/#decomposing-a-3d-matrix
Potential APIs:
Calling decompose
// From an array (2d or 3d)
Rematrix.decompose([0.5, -1, 1, 0.5, 10, -20]);
Rematrix.decompose([1.4, 0, -1.4, 0, 0.3, 2, 0, 0, 0.7, 0, 0.7, 0, 11.3, 0, -11.3, 1]);

// From a string
Rematrix.decompose("matrix(0.5, -1, 1, 0.5, 10, -20)")
Rematrix.decompose("matrix3d(1.4, 0, -1.4, 0, 0.3, 2, 0, 0, 0.7, 0, 0.7, 0, 11.3, 0, -11.3, 1)")

Potential return value shapes
// hierarchical
{
  translate: { x: 10, y: 0, z: 0 },
  rotate: { x: 40, y: -25, z: 16 },
  skew: { x: 0, y: 0 },
  scale: { x: 1, y: 1, z: 1 },
  perspective: 0
}

// as 3d vectors
{
  translate: [10, 0, 0],
  rotate: [40, -25, 16],
  // ... etc.
}

// verbose
{
  translateX: 10,
  translateY: 0,
  translateZ: 0,
  // ... etc.
}

From the above, I prefer either { x, y, z } or [x, y, z] format. I think hierarchical is more explicit but vectors is probably more useful to use in a wider range of use cases that can just accept a vector, e.g., for translate.

Calling decompose

// From an array (2d or 3d)
Rematrix.decompose([0.5, -1, 1, 0.5, 10, -20]);
Rematrix.decompose([1.4, 0, -1.4, 0, 0.3, 2, 0, 0, 0.7, 0, 0.7, 0, 11.3, 0, -11.3, 1]);

// From a string
Rematrix.decompose("matrix(0.5, -1, 1, 0.5, 10, -20)")
Rematrix.decompose("matrix3d(1.4, 0, -1.4, 0, 0.3, 2, 0, 0, 0.7, 0, 0.7, 0, 11.3, 0, -11.3, 1)")

Looks 💯 — Using Rematrix.format() and Rematrix.parse() should make it easy to accept valid CSS strings, plus both short and long form matrixes.
Potential return value shapes
The verbose model is what came to my mind first, I think because object keys then correspond to Rematrix method names (and facilitate an elegant compose() method)

const decomposed = Rematrix.decompose(...)

/**
 * decomposed === {
 * 	translateX: 10,
 * 	translateY: 0,
 * 	translateZ: 0,
 * 	... etc.
 * }
 */

function compose(decomposed) {
	return Object.keys(decomposed)
		.map(key => Rematrix[key](decomposed[key]))
		.reduce(Rematrix.multiply)
}

Take this example from the documentation:
var element = document.querySelector('#example')
var style = getComputedStyle(element)['transform']

var transform = Rematrix.parse(style)

var r1 = Rematrix.rotateZ(20)
var r2 = Rematrix.rotateZ(65)
var r3 = Rematrix.rotateZ(-40)

var product = [transform, r1, r2, r3].reduce(Rematrix.multiply)

If we re-write this using a verbose decompose() return type, and compose()...
var element = document.querySelector('#example')
var style = getComputedStyle(element)['transform']

var transform = Rematrix.decompose(style)

transform.rotateZ += 20
transform.rotateZ += 65
transform.rotateZ -= 45

var product = Rematrix.compose(transform)

It creates a layer of abstraction that gently changes the workflow, but expects less interaction with individual matrixes or transform methods. I’m not stuck on the idea, but it is growing on me... what’s your impression of this?

I think verbose is fine, you make a good point of it mapping directly to CSS property names and Rematrix methods.

So the easy parts are done haha. I took a quick look at the W3C algorithm, and it was frightening. I found some vague answers to this StackOverflow question a little helpful.
I’ll still be reachable, but I expect to be busy all of June finishing up another project. Please feel free to start exploring the guts of a decompose() method, but if not, I will gladly run with your idea when I free up.

The CSS string would be handy too.

I noticed the W3C CSS Transform Module document no longer included the 3D matrix decomposition sections. But thanks to WayBackMachine....
Input:  matrix      ; a 4x4 matrix

Output: translation ; a 3 component vector
        scale       ; a 3 component vector
        skew        ; skew factors XY,XZ,YZ represented as a 3 component vector
        perspective ; a 4 component vector
        quaternion  ; a 4 component vector

Returns false if the matrix cannot be decomposed, true if it can

// Normalize the matrix.
if (matrix[3][3] == 0)
    return false

for (i = 0; i < 4; i++)
    for (j = 0; j < 4; j++)
        matrix[i][j] /= matrix[3][3]

// perspectiveMatrix is used to solve for perspective, but it also provides
// an easy way to test for singularity of the upper 3x3 component.
perspectiveMatrix = matrix

for (i = 0; i < 3; i++)
    perspectiveMatrix[i][3] = 0

perspectiveMatrix[3][3] = 1

if (determinant(perspectiveMatrix) == 0)
    return false

// First, isolate perspective.
if (matrix[0][3] != 0 || matrix[1][3] != 0 || matrix[2][3] != 0)
    // rightHandSide is the right hand side of the equation.
    rightHandSide[0] = matrix[0][3]
    rightHandSide[1] = matrix[1][3]
    rightHandSide[2] = matrix[2][3]
    rightHandSide[3] = matrix[3][3]

    // Solve the equation by inverting perspectiveMatrix and multiplying
    // rightHandSide by the inverse.
    inversePerspectiveMatrix = inverse(perspectiveMatrix)
    transposedInversePerspectiveMatrix = transposeMatrix4(inversePerspectiveMatrix)
    perspective = multVecMatrix(rightHandSide, transposedInversePerspectiveMatrix)
else
    // No perspective.
    perspective[0] = perspective[1] = perspective[2] = 0
    perspective[3] = 1

// Next take care of translation
for (i = 0; i < 3; i++)
    translate[i] = matrix[3][i]

// Now get scale and shear. 'row' is a 3 element array of 3 component vectors
for (i = 0; i < 3; i++)
    row[i][0] = matrix[i][0]
    row[i][1] = matrix[i][1]
    row[i][2] = matrix[i][2]

// Compute X scale factor and normalize first row.
scale[0] = length(row[0])
row[0] = normalize(row[0])

// Compute XY shear factor and make 2nd row orthogonal to 1st.
skew[0] = dot(row[0], row[1])
row[1] = combine(row[1], row[0], 1.0, -skew[0])

// Now, compute Y scale and normalize 2nd row.
scale[1] = length(row[1])
row[1] = normalize(row[1])
skew[0] /= scale[1];

// Compute XZ and YZ shears, orthogonalize 3rd row
skew[1] = dot(row[0], row[2])
row[2] = combine(row[2], row[0], 1.0, -skew[1])
skew[2] = dot(row[1], row[2])
row[2] = combine(row[2], row[1], 1.0, -skew[2])

// Next, get Z scale and normalize 3rd row.
scale[2] = length(row[2])
row[2] = normalize(row[2])
skew[1] /= scale[2]
skew[2] /= scale[2]

// At this point, the matrix (in rows) is orthonormal.
// Check for a coordinate system flip.  If the determinant
// is -1, then negate the matrix and the scaling factors.
pdum3 = cross(row[1], row[2])
if (dot(row[0], pdum3) < 0)
    for (i = 0; i < 3; i++)
        scale[i] *= -1;
        row[i][0] *= -1
        row[i][1] *= -1
        row[i][2] *= -1

// Now, get the rotations out
quaternion[0] = 0.5 * sqrt(max(1 + row[0][0] - row[1][1] - row[2][2], 0))
quaternion[1] = 0.5 * sqrt(max(1 - row[0][0] + row[1][1] - row[2][2], 0))
quaternion[2] = 0.5 * sqrt(max(1 - row[0][0] - row[1][1] + row[2][2], 0))
quaternion[3] = 0.5 * sqrt(max(1 + row[0][0] + row[1][1] + row[2][2], 0))

if (row[2][1] > row[1][2])
    quaternion[0] = -quaternion[0]
if (row[0][2] > row[2][0])
    quaternion[1] = -quaternion[1]
if (row[1][0] > row[0][1])
    quaternion[2] = -quaternion[2]

return true

Example of the W3 implementation:
https://github.com/facebook/react-native/blob/master/Libraries/Utilities/MatrixMath.js#L572
But I think the Facebook implementation has bugs (line duplicates? double check quaternions convetation) and also it's not optimized.
There are a lot of unrequired arrays spawns.
Also, probably, perspective calc can be skipped?
And algorithm for the 3d matrix decomposition is moved here:
https://www.w3.org/TR/css-transforms-2/


Coverage report link is broken

Description
In README.md and CONTRIBUTING.md, the Code coverage link brings you to a codecov page that does not provide the expected coverage info. It says:

This repo has been deactivated
Contact an administrator of your git organization to grant write-permissions in your git-provider for this repository.

Related Issues
No response
Questions
No.
Demo
https://app.codecov.io/github/stdlib-js/stdlib/branch/develop
Reproduction
No response
Expected Results
No response
Actual Results
![image](https://github.com/stdlib-js/stdlib/assets/119948/0a882d33-3251-48e3-bfcd-78b7e49012fe)

Version
No response
Environments
N/A
Browser Version
No response
Node.js / npm Version
No response
Platform
No response
Checklist

[X] Read and understood the Code of Conduct.
[X] Searched for existing issues and pull requests.

@rotu Thanks for flagging! Looks like Codecov had changed how it worked with Git repos a while back and we had yet to migrate over. This should now be fixed. The links should work.
Actually generating and uploading the actual coverage report to Codecov is, however, not working at the moment. Namely because we don't run the full test suite across the entire project during CI due to the sheer number of tests the main repository contains.
I'll go ahead and close this as completed. Uploading accurate reports is a separate follow-on task.


feature: basic Javascript [Variables & Functionality]

USER STORY
AS a USER
I WANT to see basic Javascript content displayed on a page
SO THAT I can start using features in my project
ACCEPTANCE TEST FOR USER STORY
GIVEN a starter code
WHEN I open  project page
THEN I should see basic javascript content displayed in my project

Tasks performed and tested


Add Roboto and Souce Sans Pro

Description
Apart from CC Accidenz Commons, the CC sites use 'Roboto' and 'Source Sans Pro' in their UIs. Citing benefits listed in https://github.com/creativecommons/fonts/issues/1#issuecomment-548986367, it would be wise to include those fonts here as well.
Problems solved
https://github.com/creativecommons/fonts/issues/1#issuecomment-548986367
Alternatives
We are currently importing those fonts into Vocabulary from Google Fonts, which works perfectly fine although there is scope for improvement.

@dhruvkb Can I work on this?

@Tanuj22 please do.

@dhruvkb This is the stylesheet in Vocabulary https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,600,700|Roboto+Condensed. Should I define fonts for every part like cyrillic, greek etc. ?

@dhruvkb if no one is working on this, Shall I take this up?

Alright!


Password required for managed users - Integrated window authentication

Integrated Windows Authentication
If your desktop or mobile application runs on Windows, and on a machine connected to a Windows domain - AD or AAD joined - it is possible to use the Integrated Windows Authentication (IWA) to acquire a token silently. No UI is required when using the application.
I have tried to run a sample MS code of MSAL Java Integrated window authentication to acquire a token with machine connected to AAD and it throws an below exception,
ERROR com.microsoft.aad.msal4j.PublicClientApplication - [Correlation ID: 0b3328c3-0644-45bd-961c-09a45f9c851a] Execution of class com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: Password is required for managed user.
Can you help me what could be the issue?
AUTHORITY=https://login.microsoftonline.com/organizations/
SCOPE=user.read

@Avery-Dunn Thanks Avery for replying to my query.
Currently Federated users is not feasible in our current setup environment.
Is there any alternate token acquisition methods which i can use without UI intervention?
Is it good to use WAM for token acquisition method using windows login credential without UI intervention?
I see WAM supported only in .net. Is this method having any constraints and will it support in other languages in future?
Thanks once again for the support.
Regards,
Ram

@ramsay2021 Unfortunately IWA is probably the best fully non-interactive flow, but there are a couple other options:

Device code flow

If you're trying to avoid UI interaction because users may not be able to access a browser on the device the app is running on, then you could use the device code flow. Your users would be able to use another device (such as their phone) to enter their credentials

Client credentials flow

It shouldn't need any UI interaction from users, but isn't really meant for applications that are tailored to specific users. If your scenario allows you to sort users into groups and access resources based on those groups (rather than by specific user's credentials/permissions), you could use this flow

Username/password flow

If you happen to have access to your user's usernames and passwords, then you can get tokens by handling them directly. This flow isn't  recommended though, as there is an obvious risk of exposing the credentials

And for any of the flows our library supports, you should be able to use a silent token call to repeatedly refresh cached tokens, so users won't need to interact with a UI and provide their credentials every time they need to sign in.
As for WAM, we unfortunately don't support it yet but we've just begun working on it and plan to have the same level of support as .NET. However, we don't expect this to be ready for a couple of months.

Closing due to inactivity. If you still have issues or have any related questions, feel free to leave a comment or re-open this thread.


isRunning() should be false when torque map is static

This check makes the renderer produce a tile when isRunning() is true. At the moment, unless we call pause(), isRunning() returns true with -torque-frame-count: 1, which empties the tile cache, calling renderTile() unnecessarily.
Should we:

pause() on maps with one step,
Or change the check so it only invalidates the cache when steps are more than one?

@javisantana

Done in #136


Improve memory usage with Hadoop Streaming

Erlang is eager with stdio which makes it very memory intensive on Hadoop Streaming, as it reads the entire input into memory without a buffer, and before it's requested. This means that your files have to fit into memory, which is sometimes unrealistic.
This cannot be fixed inside the Vessel application code, but perhaps we could ship a binary wrapper which pipes through the input at a rate that doesn't bloat the Erlang memory usage.
This is labelled an enhancement as it's an Erlang attribute, rather than a Vessel bug. It may be that we cannot solve this effectively (in which case, just tune as mentioned in the README).

Could we do something like working with streams in Erlang? https://github.com/tim/erlang-streams or https://github.com/rpt/estreams or https://github.com/epappas/erl_streams
None of them have examples involving loading files, unfortunately.
Elixir has Streams in the stdlib and it's very convenient. Maybe that's something to explore? https://hexdocs.pm/elixir/Stream.html

@Trevoke aloha, but unfortunately it's an issue in the BEAM rather than the application layer.
Vessel already uses Streams (see here), but the issue is that the BEAM loads :stdio eagerly so even though you're streaming :stdin, it still reads the entire thing into memory on startup.
You can try that yourself by piping large IO into an Elixir archive which just sleeps (or whatever); it'll eagerly load the buffer even though you're not actually asking for it, or even interacting with it at all.
That's why this is such an annoyance to fix; you would need some sort of binding around Vessel in another language which operates as a buffer (and even then, there has to be message passing between BEAM and the binding as it can't just pipe straight through due to the same IO loading).
I was going to roll something in Rust or C which you can just request the next chunk of input with, but never got around to it as it feels like something which should stand as a separate project.
Does all of this make sense?
If you have any other ideas/suggestions/solutions/questions/etc, feel free to drop comments in here; I'm a bit lost as the best way to solve this issue, but it does need fixing at some point.

In Elixir, we can stream files (tested with a 1.9GB file, unless that loaded to memory in under 5 seconds).
Here is a simple example from the documentation of loading a file and making a change to the contents and streaming it back out to another file without loading it all into memory:
stream = File.stream!("code")
|> Stream.map(&String.replace(&1, "#", "%"))
|> Stream.into(File.stream!("new"))
|> Stream.run

Down in Erlang, we can stream down at this level

http://erlang.org/doc/man/file.html#read-2
http://erlang.org/doc/man/file.html#pread-2

For compression, apparently file:open/2 has a compressed option: http://erlang.org/doc/man/file.html#open-2

So... I don't understand why we need to be working with :stdio ?

@Trevoke
You question tells me that you're unfamiliar with the Hadoop Streaming protocol? The idea behind the protocol is that everything is done via :stdio in order to avoid rolling your own file handling (and basically rewriting Hadoop itself). We're tied to :stdio due to the protocol; using files directly is not an option.
Thanks for the info, but I am already very familiar with Streams - as previously mentioned Vessel already uses them. Streaming :stdio is very different to streaming a file in that the BEAM eagerly loads :stdin regardless of whether you're ready to use it (basically making the streaming aspect useless). This is a BEAM-ism rather than application logic behaving badly; you can see the issue by doing literally nothing in your application but piping files in and watching your memory go up and up and up.
Does that make sense?

Yeah.. It makes perfect sense. I thought we were implementing more of the
mangling process, which is why I figured the stream might make sense, I
didn't realize we were just picking them up when Hadoop gives them to us,
but that makes sense too.
Sorry about hassling you :)
On Mon, Feb 20, 2017 at 12:36 AM Isaac Whitfield<EMAIL_ADDRESS>wrote:

@Trevoke https://github.com/Trevoke
You question tells me that you're unfamiliar with the Hadoop Streaming
protocol? The idea behind the protocol is that everything is done via
:stdio in order to avoid rolling your own file handling (and basically
rewriting Hadoop itself). We're tied to :stdio due to the protocol; using
files directly is not an option.
Thanks for the info, but I am already very familiar with Streams - as
previously mentioned Vessel already uses them. Streaming :stdio is very
different to streaming a file in that the BEAM eagerly loads :stdin
regardless of whether you're ready to use it (basically making the
streaming aspect useless). This is a BEAM-ism rather than application logic
behaving badly; you can see the issue by doing literally nothing in your
application but piping files in and watching your memory go up and up and
up.
Does that make sense?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zackehh/vessel/issues/5#issuecomment-280992231, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEJSduKrUOmXpn16YbcYK1ZBN5NgmCRks5reSZHgaJpZM4K-N6w
.

@Trevoke no problemo! I think maybe I assumed that knowledge in the original issue, so my apologies if it was unclear :)


Updating schema link and users what's new for 5.1

See https://trello.com/c/bm6wvqIt/147-data-model-docs-updates

Should we hash out https://github.com/openmicroscopy/openmicroscopy/pull/3495 first?

Yes, this can wait til after m4. And is aimed at users rather than devs anyway.

Actually, given this updates the model version, it could do with going out with m4 if we get the model stuff sorted today. I'll push a fix shortly, we always have the 'not released yet' disclaimer at the top of the page to cover us ;)

How's this @ximenesuk @joshmoore ?

@joshmoore happy for me to merge this for m4 once everything is green?

LGTM.

--no-rebase


Get-OBRecoverableItem example no longer works...

The section that starts...
"You can also search for items to restore using the Get-OBRecoverableItem cmdlet. In our example, to search for finances.xls we could get a handle on the file by running this command:"
Is no longer correct because you get an error saying "WARNING: This functionality is deprecated. Please use Instant Recovery Option."
Is there an option to allow you to search/list the recoverable items and if so can you update this page to include a working example?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f8704f4e-aca9-f84e-b5f8-84c0602dc6ec
Version Independent ID: d8a0b916-91c2-49ac-b5d7-63d8755b7eb2
Content: Use PowerShell to back up Windows Server to Azure
Content Source: articles/backup/backup-client-automation.md
Service: backup
GitHub Login: @saurabhsensharma
Microsoft Alias: saurse

This is still an issue, documentation is not updated yet. Get-OBRecoverableItem cmdlet is not working. Please fix documentation.

This has been fixed and the documentation update should happen by 12/3.

#please-close


Discuss architecture of R53 backup system

Acceptance Criteria

Schedule meeting to discuss architecture
Arrive on decision regarding architecture of system

Proposed architecture:
In operations-engineering repository:

Workflow to trigger recover from backup lambda
Terraform stack using route 53 backup and restore module

In reusable route 53 backup module repository:

Terraform and logic for:
Backup lambda
Restore lambda
Scheduling system e.g. CloudWatch
S3 to store backup data

Justification:

reusable code
cloud native

Ask who wants an invite and schedule this meeting.

Agreed to follow current approach:

Backup GHA workflow on schedule
GHA workflow to restore R53 data
boto3 Python scripts
Terraform for S3 bucket


Enable eventwatcher replacement with key-pair value pattern

What would you like to be added:
Currently, the PipeCD eventwatcher value replacement only supports value under a simple query path like $.spec.xyz[1].yy... (only index and key name allowed). It would be better if we support key-value pair query as well.
Note: even better if we can support wildcard queries.
ref kubectl docs
Why is this needed:

Created issue on go-yaml repository https://github.com/goccy/go-yaml/issues/290

I close this issue because this is almost the same as [Support multiple value updates in EventWatcher] #5088


Sort order changes on double click on radio button; reversed sort works only sometimes; (strange side-effects)

use on change event instead of click:

https://stackoverflow.com/questions/6358673/javascript-checkbox-onchange
https://www.w3schools.com/jsref/event_onchange.asp


[BACKPORT to 1.5.latest] Use events.contextvar because of multiprocessing unable to pickle ContextVar (#7949)

Backport of #7949 to 1.5.latest

Sorry folks, how does the release for this work? Will it be included in 1.5.3 ?

@jelstongreen Correct!

https://github.com/dbt-labs/dbt-core/issues/7988


APNS code should check result for "Success" and propagate errors

In the below code the function _apns_send returns a dict of the token(s) and return value. It appears from my debugging session that the following are possible return values:

Success
MissingTopic
Unregistered
and more?

Right now the library only checks for Unregistered tokens but doesn't provide feedback for other types of APNS errors. I propose exposing these errors further up the stack so they can be handled in some way in application code.
def apns_send_message(registration_id, alert, application_id=None, certfile=None, **kwargs):
	"""
	Sends an APNS notification to a single registration_id.
	This will send the notification as form data.
	If sending multiple notifications, it is more efficient to use
	apns_send_bulk_message()

	Note that if set alert should always be a string. If it is not set,
	it won"t be included in the notification. You will need to pass None
	to this for silent notifications.
	"""

	try:
		result = _apns_send(
			registration_id, alert, application_id=application_id,
			certfile=certfile, **kwargs
		)
	except apns2_errors.APNsException as apns2_exception:
		if isinstance(apns2_exception, apns2_errors.Unregistered):
			device = models.APNSDevice.objects.get(registration_id=registration_id)
			device.active = False
			device.save()
		raise APNSServerError(status=reason_for_exception_class(apns2_exception.__class__))
	return result

def apns_send_bulk_message(
	registration_ids, alert, application_id=None, certfile=None, **kwargs
):
	"""
	Sends an APNS notification to one or more registration_ids.
	The registration_ids argument needs to be a list.

	Note that if set alert should always be a string. If it is not set,
	it won"t be included in the notification. You will need to pass None
	to this for silent notifications.
	"""

	results = _apns_send(
		registration_ids, alert, batch=True, application_id=application_id,
		certfile=certfile, **kwargs
	)
	inactive_tokens = [token for token, result in results.items() if result == "Unregistered"]
	models.APNSDevice.objects.filter(registration_id__in=inactive_tokens).update(active=False)
	return results

In the below code the function _apns_send returns a dict of the token(s) and return value. It appears from my debugging session that the following are possible return values:
Success
MissingTopic
Unregistered
and more?

Return values of _apns_send can be "Success" or any of these errors:
https://github.com/jleclanche/django-push-notifications/blob/master/push_notifications/apns_errors.py
What kind of error exposing would be most useful for you?

I'm not sure exactly what would fit best without breaking the existing API, but I was having trouble understanding why my requests weren't working and the only way to figure it out was to pause in the debugger and inspect the return values.
Although apns_send_bulk_message returns results, the code I copied above actually contains some modifications to capture and return the result in apns_send_message. I'll submit a PR for that and add docs about the return values.


Fixes #2033 Keyboard scrolling in completions list broken

Fixes #2033 Keyboard scrolling in completions list broken
Fixes definition of scroll viewer.

👍


GPU memory and batch size

Hello, I'm experimenting DeepFace for emotion recognition, but when I try to analyze an image I get the following exception:
ResourceExhaustedError: failed to allocate memory [Op:AddV2]
Many posts on TensorFlow state this problem is related to the batchsize used during model training and recommend reducing it. However, how to do it? The code is as follows:
import cv2
import matplotlib.pyplot as plt
from deepface import DeepFace
boy_happy = cv2.imread('image/boy_happy_1.jpg')
plt.imshow(cv2.cvtColor(boy_happy, cv2.COLOR_BGR2RGB))
predictions = DeepFace.analyze(boy_happy)

The file is a 64Kb version of the following: https://www.dreamstime.com/stock-images-portrait-happy-boy-showing-thumbs-up-gesture-isolated-over-white-background-image34772834.
Thanks in advance!

your gpu memory seems not be enough. try disable it.
impost os
os.setenv['CUDA_VISIBLE_DEVICES'] = ""
#or os.setenv['CUDA_VISIBLE_DEVICES'] = "-1"

Thank you! Indeed, my GPU has only 4Gb. Following your suggestion, in the very beggining of my Jupyter notebook I wrote:
import os
os.environ['CUDA_VISIBLE_DEVICES'] = ""
import cv2
import matplotlib.pyplot as plt
from deepface import DeepFace

However, the result was precisely the same.

It works if we do like this:
import os
os.environ['CUDA_VISIBLE_DEVICES'] = "-1"


DRAFT: First implementation output/history_points.py

Should fix #55

Is this ready or would you like to do more?

I'm going to push few more commits in this branch... And I still need to add a test!

Oops, I try to rebase to master, and I see that I don't understand session_00. What should I do with the .his file?

The .his file should be discovered by this rule:
https://github.com/exabl/snek5000/blob/7e7859c8ee732de8ed889c189008d893bc09c364/src/snek5000/assets/internal.smk#L74-L82
However since a .his file is not always present. It would be enough to detect it using glob / iglob under snakemake rule's params instead of input section. Like here in rule archive in io.smk. Then a symlink should be provided if the .his file exists.
https://github.com/exabl/snek5000/blob/7e7859c8ee732de8ed889c189008d893bc09c364/src/snek5000/util/files.py#L83-L113

Could we target Python >= 3.8 ?

Yes we can do Python 3.8+

I don't understand the issue with codecov/project. @ashwinvis do you understand ?

Also fix #75 (commit 72b3cdf1e706fed063e258fdeaf983e39721f99a)


KARAF-3596: Added xerces.stax packages to org.osgi.framework.system.pack...

...ages.extra, and removed xerces from org.osgi.framework.bootdelegation

This PR can't be merged directly (conflicts). Please reopen if required.


test: TypeOf test for simple object

Bundle size comparison

size comparison

browser
uncompressed 4.73 kB -- gzipped 1.91 kB --

main
uncompressed 4.62 kB -- gzipped 1.83 kB --

module
uncompressed 4.48 kB -- gzipped 1.83 kB --

Speed comparison
PR

library
relative speed
operations per second
avg. operation time

@typeofweb/schema@pr-branch
+2%
(1,511,478 rps)
(avg: 661ns)

@typeofweb/schema@base-branch
0%
(1,481,860 rps)
(avg: 674ns)

<EMAIL_ADDRESS>-93.03%
(103,232 rps)
(avg: 9μs)

<EMAIL_ADDRESS>-95.78%
(62,542 rps)
(avg: 15μs)

<EMAIL_ADDRESS>-96.49%
(51,979 rps)
(avg: 19μs)

<EMAIL_ADDRESS>-99.75%
(3,760 rps)
(avg: 265μs)


fix hide devices

A decision was made - given ongoing issues with Tandem on Windows - to remove Tandem from the device menu on Windows machines. When I went to go do this, I discovered that the implementation of hideUnavailableDevices was pretty problematic - located in appState instead of appActions where it logically belongs, being called during the React render cycle (surprised this wasn't causing an Invariant Violation, TBH), and mutating React state directly instead of using setState. This branch fixes these shortcomings.
This PR also includes the change to hide Tandem on Windows machines.
Ping @gniezen for review.

 [uploader] fix hideUnavailableDevices to avoid mutating state directly

Just looking back, this feature came in through #156 and it's totally my bad for not noticing/recognizing these shortcomings before. @gniezen you should take a look at our commentary there to make certain I haven't introduced any regressions; I did look, and I don't believe I have.

I built a test .zip for Nick, and he loaded it on the Windows machine in the office, and everything looked right - no Tandem, but Abbott devices present.

Pending two minor comments, LGTM and can be merged. Tested on OS X El Capitan, Windows 7 and 8.1.


ParseError: mismatched tag with generic downloader on insider.foxnews

Due diligence

[x] I've verified and I assure that I'm running youtube-dl 2<IP_ADDRESS>
[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[x] Site support request (request for adding support for a new site)

Verbose Output
carter@carterdesk:~$ youtube-dl http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words
[generic] univ-wisconsin-student-group-pushing-silence-certain-words: Requesting header
WARNING: Falling back on generic information extractor.
[generic] univ-wisconsin-student-group-pushing-silence-certain-words: Downloading webpage
[generic] univ-wisconsin-student-group-pushing-silence-certain-words: Extracting information
ERROR: Unsupported URL: http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words
carter@carterdesk:~$ youtube-dl  --verbose http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2<IP_ADDRESS>
[debug] Python version 2.7.6 - Linux-3.19.0-66-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {}
[generic] univ-wisconsin-student-group-pushing-silence-certain-words: Requesting header
WARNING: Falling back on generic information extractor.
[generic] univ-wisconsin-student-group-pushing-silence-certain-words: Downloading webpage
[generic] univ-wisconsin-student-group-pushing-silence-certain-words: Extracting information
ERROR: Unsupported URL: http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1633, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2525, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2514, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
ParseError: mismatched tag: line 115, column 2
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 691, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 347, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2377, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words

Description of your issue, suggested solution and other information
url: http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words
domain: insider.foxnews.com

Im not sure if foxnews is supposed to use a specific downloader but in the tests i see the video.foxnews.com domain but my url is from insider.foxnews.com and defaults to the generic downloader. Looks like some parse error with malformed html ParseError: mismatched tag: line 115, column 2 I bet some closing tag is missing in their html

insider.foxnews.com was not supported. I've added it and it will be available in the next version.


chore(doc): fix typos

Found via typos --hidden --format brief

in principle I think we would not merge such a PR. the typos are fixed. please feel free to contribute on issues relating to dind/stability/windows/macos.


Llama models quantized with tensor-tools don't have llama.attention.head_count in metadata

When trying to run models quantized from safetensor to gguf with tensor-tools, when trying to load a model, I get an error:
'Result::unwrap()' on an 'Err' value: Msg("cannot find llama.attention.head_count in metadata")

Right, when a model is built with tensor-tools it's to be used with a custom model rather than the quantized model that uses the metadata from the gguf file.
Not sure what your exact use case is, you can either hack tensor-tools or write your custom quantizer that injects the proper metadata file, or do this in python using candle-pyo3 (not sure if everything is exposed to do this but the goal is clearly to make it easy for such use case to be done with a couple lines of python).
Is your model supported by llama.cpp and so should work fine in candle's quantized example provided that the appropriate metadata were available?

I think the model should be supported by llama.CPP as it is a model trained by the llama.c example, but I haven't tested myself. So yes I think it would work fine if the appropriate metadata were available.

I see here a blank metadata is provided. So I could just make it and pass it here (this is in tensor-tools)
    gguf_file::write(&mut out_file, &[], &qtensors)?;

Yep that's exactly what I meant by making your custom quantizer, just hack it in there.

Thanks for the tip, I tried ripping the metadata off of another gguf llama 2 architecture gguf (it works, I tested it. Its from TheBloke), but I get
'Result::unwrap()' on an 'Err' value: Msg("cannot find tensor-infor for token_embd.weight")

I doubt there is. The error means that the model code is expecting to find a tensor named token_embd.weight but this doesn't exist in the gguf file. You can use the ls subcommand from the tensor-tools example to list all tensors in a file and maybe it's using some different naming convention there?

Ohh, I think I see my mistake. I assumed the model trained by the llama.c example would be compatible with the quantized example but I was wrong, it seems to have an entirely different structure.

Wait actually is it just a slightly different name

Nope, it is a different structure, when copying quantized_llama.rs and changing it so that the llama.c example trained model loads, the shapes are mismatched.
called Result::unwrap()on anErr value: ShapeMismatchBinaryOp { lhs: [36, 288], rhs: [1, 36, 2048], op: "reshape" }

@soupslurpr The problem you are facing with the "ec_dot_q4_0_q8_0: 9 is not even" error is that the ggml quantizations only support tensor shapes that are devidable by 8, meaning one of your weights isn't divisable by 8. You could circument this by not quantizing tensors that are not dicvisible by 8, e.g. keeping them in f16 or f32.

@LLukas22 Thanks for the advice! I'm very confident all of the numbers shown when I do tensor-tools ls on the model are divisible by 8. But the whole reason I used q4_0 and not something like q3k is because the q3k didn't quantize most of it for some reason. Anyways, using the q3k model made it successfully work! The ls for the q4_0 one is below if anyone is interested.
lm_head.weight: [[32000, 288]; Q4_0] model.embed_tokens.weight: [[32000, 288]; Q4_0] model.layers.0.input_layernorm.weight: [[288]; F32] model.layers.0.mlp.down_proj.weight: [[288, 768]; Q4_0] model.layers.0.mlp.gate_proj.weight: [[768, 288]; Q4_0] model.layers.0.mlp.up_proj.weight: [[768, 288]; Q4_0] model.layers.0.post_attention_layernorm.weight: [[288]; F32] model.layers.0.self_attn.k_proj.weight: [[288, 288]; Q4_0] model.layers.0.self_attn.o_proj.weight: [[288, 288]; Q4_0] model.layers.0.self_attn.q_proj.weight: [[288, 288]; Q4_0] model.layers.0.self_attn.v_proj.weight: [[288, 288]; Q4_0] model.layers.1.input_layernorm.weight: [[288]; F32] model.layers.1.mlp.down_proj.weight: [[288, 768]; Q4_0] model.layers.1.mlp.gate_proj.weight: [[768, 288]; Q4_0] model.layers.1.mlp.up_proj.weight: [[768, 288]; Q4_0] model.layers.1.post_attention_layernorm.weight: [[288]; F32] model.layers.1.self_attn.k_proj.weight: [[288, 288]; Q4_0] model.layers.1.self_attn.o_proj.weight: [[288, 288]; Q4_0] model.layers.1.self_attn.q_proj.weight: [[288, 288]; Q4_0] model.layers.1.self_attn.v_proj.weight: [[288, 288]; Q4_0] model.layers.2.input_layernorm.weight: [[288]; F32] model.layers.2.mlp.down_proj.weight: [[288, 768]; Q4_0] model.layers.2.mlp.gate_proj.weight: [[768, 288]; Q4_0] model.layers.2.mlp.up_proj.weight: [[768, 288]; Q4_0] model.layers.2.post_attention_layernorm.weight: [[288]; F32] model.layers.2.self_attn.k_proj.weight: [[288, 288]; Q4_0] model.layers.2.self_attn.o_proj.weight: [[288, 288]; Q4_0] model.layers.2.self_attn.q_proj.weight: [[288, 288]; Q4_0] model.layers.2.self_attn.v_proj.weight: [[288, 288]; Q4_0] model.layers.3.input_layernorm.weight: [[288]; F32] model.layers.3.mlp.down_proj.weight: [[288, 768]; Q4_0] model.layers.3.mlp.gate_proj.weight: [[768, 288]; Q4_0] model.layers.3.mlp.up_proj.weight: [[768, 288]; Q4_0] model.layers.3.post_attention_layernorm.weight: [[288]; F32] model.layers.3.self_attn.k_proj.weight: [[288, 288]; Q4_0] model.layers.3.self_attn.o_proj.weight: [[288, 288]; Q4_0] model.layers.3.self_attn.q_proj.weight: [[288, 288]; Q4_0] model.layers.3.self_attn.v_proj.weight: [[288, 288]; Q4_0] model.layers.4.input_layernorm.weight: [[288]; F32] model.layers.4.mlp.down_proj.weight: [[288, 768]; Q4_0] model.layers.4.mlp.gate_proj.weight: [[768, 288]; Q4_0] model.layers.4.mlp.up_proj.weight: [[768, 288]; Q4_0] model.layers.4.post_attention_layernorm.weight: [[288]; F32] model.layers.4.self_attn.k_proj.weight: [[288, 288]; Q4_0] model.layers.4.self_attn.o_proj.weight: [[288, 288]; Q4_0] model.layers.4.self_attn.q_proj.weight: [[288, 288]; Q4_0] model.layers.4.self_attn.v_proj.weight: [[288, 288]; Q4_0] model.layers.5.input_layernorm.weight: [[288]; F32] model.layers.5.mlp.down_proj.weight: [[288, 768]; Q4_0] model.layers.5.mlp.gate_proj.weight: [[768, 288]; Q4_0] model.layers.5.mlp.up_proj.weight: [[768, 288]; Q4_0] model.layers.5.post_attention_layernorm.weight: [[288]; F32] model.layers.5.self_attn.k_proj.weight: [[288, 288]; Q4_0] model.layers.5.self_attn.o_proj.weight: [[288, 288]; Q4_0] model.layers.5.self_attn.q_proj.weight: [[288, 288]; Q4_0] model.layers.5.self_attn.v_proj.weight: [[288, 288]; Q4_0] model.norm.weight: [[288]; F32] rot.freq_cis_imag: [[256, 24]; F32] rot.freq_cis_real: [[256, 24]; F32]

I really want to be able to quantize it with it working since the q4_0 was 14 MB but the q3k is 91 MB. But its really cool I got it running in the first place.

I think if you try to quantize it with q3k it isn't quantized at all, since the k-quants need their tensors to be devisible by 256 and i believe tensors that aren't divisible by that will simply be skipped and "quantized" to f16 or f32. Thats why your model is 4 times the size of the q4_0 version.

Oh, I see. The model unfortunately is producing trash, at least compared to when I run it from the llama2-c example. Even though I "quantized" it to f32 (it was already in f32).

Could you point me at the model that you're trying to quantize? I can give it a quick spin just to see if there is anything obvious (and also hopefully to help with this workflow though longterm I hope it's the kind of thing that we can do with the pyo3 api that @LLukas22 is putting together).

I did look into those, but they are too big in size, so I was hoping this could be smaller

Thank you so much for that, it works great. Btw, q4_0 still fails with the same not even issue for some reason

Also, I may be wrong but isn't the config already configurable? I can simply change the values in the Config struct. Or were you talking about something else?

Btw, q4_0 still fails with the same not even issue for some reason

Right, I think this one is also not needed as we don't support avx512 at the moment (and probably won't support it until it's stabilized), so I've made #1202 to remove it. Give it a spin and let me know how it gets.

Also, I may be wrong but isn't the config already configurable? I can simply change the values in the Config struct. Or were you talking about something else?

It is configurable in the api but not in the command line example. When using a binary file, the config comes from the binary file, when using safetensors/gguf the config is assumed to be tiny here.

Right, I think this one is also not needed as we don't support avx512 at the moment (and probably won't support it until it's stabilized), so I've made #1202 to remove it. Give it a spin and let me know how it gets.

It works :D

It is configurable in the api but not in the command line example. When using a binary file, the config comes from the binary file, when using safetensors/gguf the config is assumed to be tiny here.

Could it have an option to store a gguf instead of a safetensors so that the correct metadata could be included and loaded?

Could it have an option to store a gguf instead of a safetensors so that the correct metadata could be included and loaded?

@soupslurpr GGUF can store Key-Value pairs in its metadata section, meaning you could store the entire config in your gguf file with your weights, resulting in a single file you need to distribute. But you probably have to write code to load these metadata fields into candles config struct.

I think we already have code that would read this in the quantized example, the nice thing with adding the data as part of the tensor-tools quantized command is that then the files could be used with this example without additional flag but also probably with llama.cpp.

Yeah that's what I thought, it would be easier to use

That's actually harder to do than what I would have thought as when quantizing we don't have the proper config any more. So what I just did is in the example, when using .gguf weights, the config is inferred from the shape of the weights. Some information is missing at this stage so this would only work if the config is one of the variant that gets used in Karpathy's repo, see #1208 for the actual change.

That sounds pretty good, I'll try it later. Thank you again 😊. Should I close this issue now?

Just tried and it works perfectly 💯 . I'll close this issue now as everything seems resolved :D


Players respawn in the dimension instead of the overworld

Since it is currently hard to make a teleporter while in the dimension, players might get stuck in the dimension

fixed in the next patch