[QUESTION] Use UploadFile in Pydantic model

Description
Is it possible to use UploadFile in a Pydantic model? The FastAPI docs say “FastAPI's UploadFile inherits directly from Starlette's UploadFile, but adds some necessary parts to make it compatible with Pydantic and the other parts of FastAPI.”
from fastapi import FastAPI, File, UploadFile
from pydantic import BaseModel

class PydanticFile(BaseModel):
    file: UploadFile = File(...)

app = FastAPI()

@app.post("/uploadfile/")
async def create_upload_file(file: PydanticFile):
    return {"filename": file.filename}

However, UploadFile does not seem to be compatible with Pydantic. The code above – derived from the example in FastAPI's Request Files tutorial – raises a value error:
ValueError: Value not declarable with JSON Schema, field: file type=UploadFile required

I'm a FastAPI/Starlette noob so take what I'm saying with a grain of salt, but I don't think what you are trying to do really makes sense.
When you are specifying a UploadFile I believe you are telling FastAPI/Starlette that you will be providing data in a multipart form body.  Pydantic is for describing the expected JSON format body.  So trying to combine the two doesn't really make sense because you need to do one or the other.
I could be off base though ... you can wait for someone else to chime in.

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

When it says "adds necessary parts to make it compatible with Pydantic", it doesn't mean with pydantic BaseModel instances. FastAPI makes deep use of a lot of pydantic internals during the request-building process, and I believe the modifications to the type are to make it compatible with that. In particular, it is used in a way that doesn't require it to be able to live as a field on a BaseModel.
Generally, you should only use BaseModel instances in FastAPI when you know you want to parse the model contents from the json body of the request.
If you want to make use of UploadFile as an attribute of a dependency class you can, it just can't be a pydantic model. (You can look at how the auth dependencies are implemented for an example along those lines.)

This seems to be working, and maybe query parameters would ultimately make more sense here.

I would go the more RESTful approach and use /author/{author name or id}/language/{language name or id}/file.  Upload is kind of implied with the POST request.  With that said, this discussion is no long Fast API based so we should probably discontinue that discussion here.
Good day and good luck ...

Thanks a lot @michaelschmit and @dmontagu.

Thanks for the help here everyone! :clap: :bow:
Thanks for reporting back and closing the issue @laeubli  :+1:

Hello,
So, what is the best approach when working with request bodies (represented as Pydantic models) and UploadFiles in FastAPI? How can could I send the request body through multipart/form-data?
I'm pretty sure I am missing concepts about general HTTP requests, but if someone has a quick answer I'd be awesome.
Thanks in advance.

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

That's right. That example is not valid.

@perezzini if you are receiving JSON data, with application/json, use normal Pydantic models.
This would be the most common way to communicate with an API.
If you are receiving a raw file, e.g. a picture or PDF file to store it in the server, then use UploadFile, it will be sent as form data (multipart/form-data).
If you need to receive some type of structured content that is not JSON but you want to validate in some way, for example, an Excel file, you would still have to upload it using UploadFile and do all the necessary validations in your code. You could use Pydantic in your own code for your validations, but there's no way for FastAPI to do it for you in that case.

Thanks for replying back @tiangolo
And congrats on your great job in FastAPI!

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

That's right. That example is not valid.

Hi, Did you find a solution to pass UploadFile and a custom PydanticModel to a Request? I' currently stuck with the same validation Error (422)

Hello everyone,
I would also like to find a solution for this, my use case will be to send documents to the ML model in production to make and make predictions over those. But the input could vary between document files such as .pdf, .word, etc. and also the possibility to pass plane text within a .json.
Ideally, I would love a query parameter named input to switch between these types so I wouldn't have to declare different endpoints, for .json input and other for documents.
Any help would be greatly appreciated.
Best regards

Came here from Goggle, and i found this on SO which could help solve this.

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

That's right. That example is not valid.

Hi, Did you find a solution to pass UploadFile and a custom PydanticModel to a Request? I' currently stuck with the same validation Error (422)

You cannot do that. See https://github.com/tiangolo/fastapi/issues/285#issuecomment-498368734

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

That's right. That example is not valid.

Hi, Did you find a solution to pass UploadFile and a custom PydanticModel to a Request? I' currently stuck with the same validation Error (422)

You cannot do that. See https://github.com/tiangolo/fastapi/issues/285#issuecomment-498368734

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

That's right. That example is not valid.

Hi, Did you find a solution to pass UploadFile and a custom PydanticModel to a Request? I' currently stuck with the same validation Error (422)

I had the same problem. Unfortunately, you cannot do that. It's a HTTP limitation. See #285 (comment)

Yes, it's a HTTP limitation; not an issue regarding FastAPI.
I think this issue should be closed.

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

Hello,
This is giving an validation Error (422)

That's right. That example is not valid.

Hi, Did you find a solution to pass UploadFile and a custom PydanticModel to a Request? I' currently stuck with the same validation Error (422)

I had the same problem. Unfortunately, you cannot do that. It's a HTTP limitation. See #285 (comment)

Yes, it's a HTTP limitation; not an issue regarding FastAPI.
I think this issue should be closed.

@perezzini if you are receiving JSON data, with application/json, use normal Pydantic models.
This would be the most common way to communicate with an API.
If you are receiving a raw file, e.g. a picture or PDF file to store it in the server, then use UploadFile, it will be sent as form data (multipart/form-data).
If you need to receive some type of structured content that is not JSON but you want to validate in some way, for example, an Excel file, you would still have to upload it using UploadFile and do all the necessary validations in your code. You could use Pydantic in your own code for your validations, but there's no way for FastAPI to do it for you in that case.

But in case of image, how will you send image from client side if we will not use pydantic model ??

Is it possible some thins like this ?
class ocrDetection(BaseModel):   URL:Optional[str]   Image:Optional[UploadFile] = File(...)
to achieve using pydantic

Thanks a lot for your helpful comment. So I guess I'd have to explicitly separate the file from the JSON part of the multipart form body, as in:
class Properties(BaseModel):
    language: str = None
    author: str = None

@app.post("/uploadfile/")
async def create_upload_file(file: UploadFile = File(...),
                             properties: Properties):
    return {"filename": file.filename}

This seems to be working, and maybe query parameters would ultimately make more sense here.

If yo do so both will be required but i need something like both has to be optional,any help.
TIA

error:  non-default argument follows default argument
async def create_upload_file(file: UploadFile = File(...),properties: Properties):
                             ^

SyntaxError: non-default argument follows default argument

FastAPI can't handle UploadFile and JSON body together, it's very inconvenient.
E.g. to get data from HTML form and send email with attached files.
Any solutions? I have to get them, but can't 😞

Formdata
On Thu, 25 Nov, 2021, 3:34 PM Kirill, @.***> wrote:

FastAPI can't handle UploadFile and JSON body together, it's very
inconvenient.
E.g. to get data from HTML form and send email with attached files.
Any solutions? I have to get them, but can't 😞
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tiangolo/fastapi/issues/657#issuecomment-979052049,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APWBSLXOO53LVQVMFKKZZKTUNYCY7ANCNFSM4JFI7ZBA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@rudy-haryanto
https://www.freecodecamp.org/news/formdata-explained/
And same accept it on FastApI side.


Add jenkins pipeline metrics ingestion

I think the executor is covered by my integration test iirc? Thanks for
review
On Tue, 11 Oct 2022 at 08:32, Linus Norton @.***> wrote:

@.**** commented on this pull request.
In sonar-project.properties
https://github.com/hmcts/dtsse-dashboard-ingestion/pull/35#discussion_r991914637
:

sonar.javascript.lcov.reportPaths=coverage/lcov.info
-sonar.exclusions=src//*.test.ts,src//.integration_test.ts,src/main/query/.ts,src/main/run.ts
+sonar.exclusions=src//*.test.ts,src//.integration-test.ts,src/main/query/.ts,src/main/run.ts,src/main/executor.ts,src/main/jenkins/*.ts

we should add a unit test for executor. I will do it in a separate PR
—
Reply to this email directly, view it on GitHub
https://github.com/hmcts/dtsse-dashboard-ingestion/pull/35#pullrequestreview-1136908682,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AATFQJ7QJRP6UYDQ2AL4UGTWCUJ2DANCNFSM6AAAAAARBNJ5YI
.
You are receiving this because you authored the thread.Message ID:
@.***>


Edges in minimap

Edges with minimap
Currently minimap only show nodes. It would be better to show edges as well or suggest some ways how to add edges.

Maybe we will create a canvas based minimap at some point but that is not for sure. If so I could imagine that we would add an option to show edges too. At this point we won't add such an option. The only way for you to show edges is to create your own minimap component. For that you could take the existing one as a starter and edges by yourself.


[Request for Protected vars] page 7326 "Whse. Phys. Invt. Journal" - SerialNoEditable & LotNoEditable

protected var
    ItemDescription: Text[100];
//A108F0000.Event18.03 protected var
    [InDataSet]
    SerialNoEditable: Boolean;
    [InDataSet]
    LotNoEditable: Boolean;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


chore(deps): bump tokio

noticed flaky network test timeout test in #2157
caused by tokio bump, looks like the tokio::Interval is either degraded or just flaky in new release 1.27
I tracked this down to poll_ready does not wake up the task for some reason, maybe the interval is too small now?
didn't find anything in release notes. will try to reproduce and perhaps file an issue, but this change increases the wait period so it will get woken up again and the test passes as before.

Codecov Report

Merging #2162 (3a9a95e) into main (cf18edb) will decrease coverage by 54.73%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff             @@
##             main    #2162       +/-   ##
===========================================
- Coverage   71.77%   17.04%   -54.73%
===========================================
  Files         445      445
  Lines       54030    54032        +2
===========================================
- Hits        38778     9211    -29567
- Misses      15252    44821    +29569

Flag
Coverage Δ

integration-tests
17.04% <0.00%> (-2.27%)
:arrow_down:

unit-tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

crates/net/network/src/session/active.rs
25.29% <0.00%> (-59.30%)
:arrow_down:

... and 317 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add devcontainer/codespace config

This should allow contributors to use GitHub codespaces to contribute to onefetch.

Thanks @spenserblack,
As a next step, we may need to add a few things to have fully self-sufficient dev container template for onefetch, I'm thinking of the tooling for onefetch.dev: node, npm, svelte extensions...

We may need to add a few things to make it a more "self-sufficient" dev container, I'm thinking of the tooling for onefetch.dev -> node, npm, svelte extensions...

Great idea! I still have a lot more to learn, but I think it's possible to have multiple dev container configurations per repo. So potentially one can have a codespace for onefetch, and one for onefetch.dev.

Oh, BTW, might want to check out the prebuild setting.
More reading: https://docs.github.com/en/codespaces/prebuilding-your-codespaces

So potentially one can have a codespace for onefetch, and one for onefetch.dev.

That's right, I'll create a  separate dev container config under docs/vercel with the right tooling.

Oh, BTW, might want to check out the prebuild setting.

I'll definitely set that up 👍

Does the simplified devcontainer (https://github.com/o2sh/onefetch/commit/50bb474a98a41f4b0b601792af4f87fc2a9d08ed) work for you? For me it fails on libz-sys
...
--- stderr
  thread 'main' panicked at '
  failed to execute command: No such file or directory (os error 2)
  is `cmake` not installed?

Yes, I'm really sorry about that, I should have tested it before pushing it --> I just reverted it 7cc7db5
BTW, another issue when running cargo run
Error: config value 'safe.directory' was not found; class=Config (7); code=NotFound (-3)

We may need to add "git config --global --add safe.directory /workspaces/onefetch" to the Dockerfile

We may need to add git config --global --add safe.directory /workspaces/onefetch to the Dockerfile

I think that's probably best put in a postCreate command. Probably don't need an image layer for a config change.
// .devcontainer/devcontainer.json
// ...
"postCreateCommand": ".devcontainer/postCreate.sh",
// ...

#!/bin/bash
# .devcontainer/postCreate.sh
git config --global --add safe.directory $PWD

I should have tested it before pushing it

I didn't know this at first (which is why I was pushing everything to my fork's main branch), but you can create codespaces from configs on other branches. You just need to have switched to that branch in the web UI before creating the new workspace. Seems so simple in hindsight :laughing:

Thanks for the help, I think  we can skip the shell script and put the command directly in the .json file c9e8223

FYI: https://github.com/orgs/community/discussions/39697
One thing I noticed is that prebuilds count against your storage. Perhaps you want to disable and delete prebuilds (knowing that they're billable, they only seem worth it for orgs TBH :shrug:)

One thing I noticed is that prebuilds count against your storage.

I guess that explains all the notifications I was getting from GH telling me to increase my Codespaces storage usage 😅
I'll delete them, thanks for letting me know @spenserblack


Documentation for Publish API needs extra info

Looking at the documentation at: https://lbryio.github.io/lbry/ for "publish" API (and CLI, presumably) the 'name' field apparently has limitations on what characters can be in the field, but these are not documented. I have discovered through trial and error that at least spaces and commas are not allowed.  I don't know yet if other characters are also disallowed.

After digging through the code in lbry/Daemon.py and lbryschema/uri.py I finally found the regular expression governing names is:
valid_name_char = "[a-zA-Z0-9-]"
That is pretty limiting and really should be documented.

Issue moved to lbryio/lbry #1268 via ZenHub


Added microsecond support for JSON serialized DateTime objects

When using the JsonFormatter, all records are encoded using json_encode(). For the timestamp this would mean that a DateTime object results in something like:
    "datetime": {
        "date": "2014-07-31 11:15:22",
        "timezone_type": 3,
        "timezone": "Europe/Amsterdam"
    }

This way we 'loose' the ability to format the timestamp with microsecond precision. A way to avoid this loss of detail is to extend the native DateTime class with one that implements JsonSerializable which appends the microseconds.
Note that due to https://bugs.php.net/bug.php?id=60302, this means we can no longer use DateTime::createFromFormat(), so I isolated it in a private function.

Might/should this PR be reconsidered? I noticed the required PHP version has been increased to 7 :smile:

@holtkamp Take a look at the updated Logger.php, it appears to handle this with lines 316-320. I can't tell from your code that you are doing more that that.

@ZergMark , you are totally right, great! Closing issue :+1:

We do handle microsecond datetimes, except we don't format them with microseconds in json outputs. Not sure if we should by default or not. I'm leaning towards disabling the microsecond precision by default as it's a bit faster to create datetimes then. The formatting of json dates should just be configurable as well then ideally so if you enable microsecond you get it everywhere. I'll open a new issue.

@Seldaek , aah, I checked the JsonFormatter, but missed the part that the microsecond formatting gets dropped. Thanks for being sharper than me this morning :smile: Good to discuss this in a separate issue.


Template - Customers - Import from CSV

Make an endpoint /api/customers/import whereby you upload a CSV file containing customers and the application processes it as follows.
The CSV file should have: id, firstName, lastName

If id is empty, then that means you need to create a new customer
If id is not empty, then that means you need to update an existing customer

MediaType.parseMediaType is not working (build error)


Please switch to appcenter.ms instead of hockeyapp.net!

Please switch to appcenter.ms instead of hockeyapp.net to be appropriate to the repository name!

Sorry, it seems migrated to appcenter already!


fix(graph): add edge to map relationship between declarations and fields

Closes #

Changes

In creating a connected graph, we observe that the relationship between declarations and fields are not reflected as expected. Specifically, when a field has a decorator, the associated decorator is not included in the filtered model manager, leading to an error due to failed decorator validation.
This change add the edge to reflect the relationship between the parent declaration and it's fields.
Author Checklist

[x] Ensure you provide a DCO sign-off for your commits using the --signoff option of git commit.
[ ] Vital features and changes captured in unit and/or integration tests
[ ] Commits messages follow AP format
[ ] Extend the documentation, if necessary
[ ] Merging to main from fork:branchname

I think we can divide this into a couple of sub-tasks:
Create the edges from a Decorator:

Edge from the Decorator to its declaration (if present)
Edges from a Decorator declaration (if present) to all the types that are referenced by its arguments

Create the edges from the elements that can have decorators applied:

Namespace
Declarations (Enum, Concept, Scalar, Map)
Enum Values
Concept Properties
Map Keys and Values

We don't currently represent properties as nodes in the graph, so I think that decorators applied on Enum Values, Concept Properties, Map Keys and Values will need to be "hoisted" onto their parent declaration (taking care to remove duplicates!). I.e. if I have something like:
concept pii extends Decorator {}

concept Person {
   @pii
   o String name
}

We will create a graph with 2 nodes, one for the Person concept and one for the declaration of the pii decorator, with a single edge from Person to pii.
If we have this, however:
concept Category {}

concept Person {
   @pii(Category)
   o String name
}

I would expect a graph with 2 nodes, one for Person and one for Category, with an edge from Person to Category. The pii decorator does not appear in the graph because it was not declared.
I would start by creating some test model files for these scenarios and making sure that the tests fail given the current code. I'd then start to refine the code until the tests pass.

I don't understand how the functionality is split between ConcertoGraphVisitor and its base class DiagramVisitor. If I've understood this correctly the DiagramVisitor is writing PlantUML output to the FileWriter and then we specialise that to build a graph.

I take responsibility for this one. ...
DiagramVisitor was originally less diagram specific, and was a helpful default visitor implementation so that the specialized visitors could avoid duplicating the tree navigation logic. I agree that it would make sense to further hoist the navigation logic, and specialise the Diagram visitor from it.


SD-WAN Setup

Wanted to reopen https://github.com/kentik/ktranslate/issues/719 but opened new issue:
Thank you, we are trying to monitor about 6400 devices and they are static ip'ed so after the discovery we could remove the ranges if that would help. Is it a requirement that the container be on the LAN with devices? We are trying to monitor them over sd-wan connections with avg 30msec latency, how big of an issue could that be? Is there a way to know if that is causing the NULL value issues?

No requirement to be on the same lan, 30msec should be fine. Since everything is static maybe just skip the discovery process entirely? Generate the yaml for each device yourself and then just run ktrans for doing the snmp polling? I'd recommend sticking to around 50 devices per CPU core you have available.  Can also ask on https://discord.com/channels/887507000722882570/888243620367458374 and see if any other practitioners have advice.


feat: add solution to human-readable-number task

feat: add solution to human-readable-number task

.


multi-playback with same pipe object

Required Info

Camera Model
D400

Firmware Version
5.9.14

Operating System & Version
Win 10

Platform
PC

SDK Version
2.14.0

Language
C++

Segment
others

Issue Description
Hello, I'm having trouble running the following simple multi-playback sample code that fails with a nullptr exception on the second call to pipe object's start() method:
void main()
{
	rs2::pipeline pipe;

	{
		rs2::config config;
		config.enable_device_from_file( "file1.bag" );
		pipe.start( config );
		pipe.stop();
	}

	{
		rs2::config config;
		config.enable_device_from_file( "file2.bag" );
		pipe.start( config );
		pipe.stop();
	}
}

The same sample code using a pipe object managed with a smart pointer doesn't crash:
void main()
{
	std::shared_ptr<rs2::pipeline> pipe;

	{
		rs2::config config;
		config.enable_device_from_file( "file1.bag" );
		pipe = std::make_shared<rs2::pipeline>();
		pipe->start( config );
		pipe->stop();
	}

	{
		rs2::config config;
		config.enable_device_from_file( "file2.bag" );
		pipe = std::make_shared<rs2::pipeline>();
		pipe->start( config );
		pipe->stop();
	}
}

So my question is : is the pipe object not reusable for playback? is it a good/mandatory practice to manage it by pointer and reallocate it for each new playback start?
Thx for your help,
Albert

Looks to me like implementation details leaking into public API. Should not be the case. We will try to fix in one of the upcoming releases.

Should be resolved in  v2.16.1


Revert "[MM-50475][MM-50775] - Post component fixes (#12265)"

Reverts mattermost/mattermost-webapp#12272

/release-note-none


simple operations on Tensor from java?

on a tensor, and what i have in mind is a homogenous float tensor, is there a way to, say, set or get a value at a given index from java?  Or to call gemm?   i haven't been able to find anything

You can change a value at a given index in Java Tensors by using getSubtensor (on the block containing this element), change it in temporary buffer and release this buffer back to Java Tensor.
Assume you have n-dimensional tensor and you need to set value to position (p_0, p_1, …, p_(n-1)). Then you can do the following:
long fDims[] = {p[0], p[1], ..., p[n - 2]};
FloatBuffer dataFloatSubtensor = FloatBuffer.allocate((int) 1);
dataTensor.getSubtensor(fDims, p[n - 1], 1, dataFloatSubtensor);
dataFloatSubtensor.put(0, value);
dataTensor.releaseSubtensor(fDims, p[n - 1], 1, dataFloatSubtensor);
Another option: if you create homogeneous tensor on top of the Java Array, you can modify elements directly in this Array and it will be automatically passed to the tensor.


Add context cancel check when decoding cached postings

[ ] I added CHANGELOG entry for this change.
[ ] Change is not relevant to the end user.

Changes
We found out that it might take a few seconds (10s+) when decoding a large amount of postings fetched from cache. If context is already canceled (fetching from cache can be time consuming), it is unnecessary to decode all fetched postings and we can just return.
Verification

Thanks


Posting from one Razor Page to another results in a 400 Bad Request

I noticed that the Razor Pages template that includes authentication makes use of a controller for logging out rather than, say, a logout.cshtml (e.g. https://github.com/DamianEdwards/RazorPagesSample/blob/master/RazorPagesWebApplication/Pages/Account/Logout.cshtml). I tried to change it to use a logout.cshtml page and discovered that posting from one page to another (setting the "action" attribute of a form) results in a 400 status code when the form is submitted.
I am interested in learning if this is by design, and if so, why? Does it have something to do with AntiForgery checks, which I appreciate is disabled if you explicitly set the action on a form element?

This is almost certainly a missing antiforgery token (logs would confirm that). Razor Pages always validates an antiforgery token on a POST.
If you're using the form tag helper, you can use asp-page="/some/other/page" for linking rather than action.

Thanks.
Do you know why a controller was included in the template?

I assume that you're talking about this one: https://github.com/aspnet/Templates/blob/dev/template_feed/Microsoft.DotNet.Web.ProjectTemplates.2.0/content/RazorPagesWeb-CSharp/Controllers/AccountController.cs
I'm actually not sure. @danroth27 do you know? Is this a mistake?

There's no UI for the logout action, so a page doesn't really make sense. We decided to use controller actions for endpoints that have no associated UI.
@davidfowl @DamianEdwards

Yep, this was by design.


CP-19612: Update cloudzero-agent chart to include region parameter

Same as https://github.com/JonParsons11350/cloudzero-charts/pull/1
Don't look at this yet - should have been in draft
By submitting a PR to this repository, you agree to the terms within the CloudZero Code of Conduct. Please see the contributing guidelines for how to create and submit a high-quality PR for this repo.
Description
This change adds "region" as a required input parameter. This is necessary to uniquely identity clusters in CloudZero - since the same cluster name + cloud account ID could be deployed in multiple regions, region is necessary to distinguish between the two.
References

Include any links supporting this change such as a:

GitHub Issue/PR number addressed or fixed
StackOverflow post
Support forum thread
Related pull requests/issues from other repos

If there are no references, simply delete this section.

Testing

Describe how this can be tested by reviewers. Be specific about anything not tested and reasons why. If this library has unit and/or integration testing, tests should be added for new functionality and existing tests should complete without errors.
Please include any manual steps for testing end-to-end or functionality not covered by unit/integration tests.
Also include details of the environment this PR was developed in (language/platform/browser version).

[ ] This change adds test coverage for new/changed/fixed functionality

Checklist

[ ] I have added documentation for new/changed functionality in this PR
[ ] All active GitHub checks for tests, formatting, and security are passing
[ ] The correct base branch is being used, if not main

Closing in lieu of https://github.com/Cloudzero/cloudzero-charts/pull/43


Fix property revert logic in the sectioned inspector

Fixes https://github.com/godotengine/godot/issues/65106.

Thanks!


LOCAL_MODE_TENANT_ID documentation update

Documentation update to ensure LOCAL_MODE_TENANT_ID is used for local mode instead of "local"

Thanks for jumping on this doc!

thanks for taking a pass 😀


Pylance doesn't understand chain of nested wildcard imports

Environment data

Language Server version: Pylance language server 2021.12.2 (pyright 211ebe44)
OS and version: Ubuntu 18.04
Python version (& distribution if applicable, e.g. Anaconda): 3.8

Expected behaviour
I'm demonstrating this issue based on librosa. Librosa's main __init__.py uses a standard symbol forwarding strategy. For instance, it re-exports the symbols from its core sub-module:
from .core import *  # pylint: disable=wildcard-import

The core sub-module in turns uses a similar approach to re-export symbols.
It looks like pylance doesn't properly understand the chain of re-exports. Even though e.g. stft is a function that is in scope of librosa itself (at runtime), it doesn't understand that.

Note however, that it still does see the stft symbol when imported through the core package:

This could indicate that the issue is caused by the chain of wildcard re-exports.
I also tried to fix the behavior by using the __all__ = [_ for _ in dir() if not _.startswith("_")] technique in the main __init__.py but it does not seem to make a difference.
Actual behaviour
The librosa.stft symbol should be understood by pylance exactly the same as librosa.core.stft.
Logs
(my logs didn't contain much meaningful)
Starting Pylance language server.

Code Snippet / Additional information
import librosa
import librosa.core

librosa.stft
librosa.core.stft

The problem is that librosa is using a dynamic expression to define __all__. In librosa/core/__init__.py, you'll find the following statement:
__all__ = [_ for _ in dir() if not _.startswith("_")]

Static type checkers like the one used in pylance don't execute the Python code; they need to be able to statically evaluate the code. Authors of Python static type checkers have agreed upon a limited set of expression forms that can be used to define __all__. For a fully-documented list, see this documentation.
If I replace the above definition of __all__ with the following supported forms, it works fine.
__all__ = []
__all__.extend(convert.__all__)
__all__.extend(audio.__all__)
__all__.extend(spectrum.__all__)
__all__.extend(pitch.__all__)
__all__.extend(constantq.__all__)
__all__.extend(harmonic.__all__)
__all__.extend(fft.__all__)
__all__.extend(notation.__all__)

If this is a library you depend upon, I recommend contacting the library's maintainers and making them aware of this issue to see if they are willing to address it in their next update.

The problem is that librosa is using a dynamic expression to define all. In librosa/core/init.py, you'll find the following statement:

I was indeed wondering whether these dynamic __all__ specifications might be an issue, but why are they working then for librosa/core/__init__.py in the first place? The fact that they work there indicated that pylance seems to do something to resolve them, but it only doesn't work across multiple re-export stages.

This doesn't have anything to do with multiple re-export stages. It will happen any time a wildcard import references a module that defines __all__ in a way that pylance doesn't understand.
The librosa/__init__.py file uses the statement from .core import *. That's a wildcard import, so only the symbols specified in the __all__ statement in librosa/core/__init__.py are included in this import. Pylance is assuming that __all__ reference no symbols because it can't understand the dynamic statement.
I think we could handle this situation better than we currently do. If we detect an unsupported statement that manipulates __all__, we should fall back to assuming that all symbols are included in a wildcard statement rather than assuming none of them are.
I've implemented this change, and it "fixes" the problem you've reported, although it would still be preferable for the package author to use a form of __all__ statement that is supported by static type checkers.
This change will be included in the next release of pylance.

This issue has been fixed in version 2022.1.0, which we've just released. You can find the changelog here:  CHANGELOG.md


Link to the participation/submission page from exercise scores page

If instructors / tutors want to inspect the number of submissions and results in the exercise scores page, they have to navigate to the participations page first. Then they need to find the students and can click on the page (e.g. participation/{participationId}/submissions)
It would be much easier, if instructors / tutors could click on the name of the student in the exercise scores page to navigate to the participation/submission page

Implemented in #1046.


:GoInstallBinaries installs without .exe on Windows with GVIM

Did the following:

Installed Go freshly on a new laptop with the official msi (Windows 10 x64)
Installed vim-go with plug on GVIM 8.1
Ran :GoInstallBinaries in gvim

all binaries installed don't end with .exe resulting in vim-go features not functioning.
Running something like go get -u github.com/mdempsky/gocode from cmd seems to do the job as expected.

Wrong repo, original post is now over here


php support for tools

Is there a simple way to support adding php tools?

Sorry, We will not support any other languages.


Tf tools renaming

(draft for now, to trigger compilation across platforms...)

I'd still recommend to keep the tensorflow- prefix...

I'd still recommend to keep the tensorflow- prefix...

Hi @saudet , the whole purpose of renaming tensorflow-tools to something else is precisely for decoupling it from TensorFlow, since it has no dependency on it. So I think removing that prefix is the first thing to do to go in that direction.

If that's the case, then the package name should also be changed.

If that's the case, then the package name should also be changed.

Not necessarily since the artifact is still under the org.tensorflow group. If we move it to a different repo, then we can replace that package name as well.

Ok this PR is ready to be reviewed and merged. Of course, scrutinizing all impacted files would be a dreadful task so I'm summarizing the changes here:

Rename org.tensorflow:tensorflow-tools artifact to org.tensorflow:ndarray
Renaming packages

Renaming package org.tensorflow.tools.ndarray to org.tensorflow.ndarray
Moving org.tensorflow.tools.Shape under org.tensorflow.ndarray
Moving org.tensorflow.tools.buffer.* under org.tensorflow.ndarray.buffer
Moving org.tensorflow.tools.buffer.impl.* under org.tensorflow.ndarray.impl.buffer

Upgrading version 0.1.0-SNAPSHOT to 0.2.0-SNAPSHOT (since backward compatibility is severely broken)
Update documentation links and examples (javadoc + .md)

Like explained previously in this thread, the purpose of this change is to decouple the previous tensorflow-tools library from TensorFlow by renaming it (as it does not depend on it). The artifact is still distributed (for now) under the org.tensorflow group, until we decide (and can) move it to another repository.

The prefix thing is just a nice convention, nothing more. If you leave out the tensorflow- prefix and decide to create a submodule in it, what are you going to name it? We should think a bit further down the road about these things so we don't end up renaming everything all the time.

What about the following convention:

group name = "org.tensorflow"
artifact name = package name without "org.tensorflow", replacing all '.' with '-'

We'd get the following directory structure:

core

core-api
core-generator
core-platform
core-platform-mkl
core-platform-gpu
core-platform-mkl-gpu

framework
ndarray

I don't really like it because we end up with JAR files named "core.jar" and "ndarray.jar", but it's better than naming modules randomly without following a convention like you're proposing.

but it's better than naming modules randomly without following a convention like you're proposing.

I don't agree, in your last example, by removing the tensorflow- prefix, we lose completely track of which artifacts is actually built around TensorFlow and which is not (which is again the purpose of this PR). But I see where you are going with a more strict naming convention. Just to recall, we have started from a project (TF1.x) that has no convention as well for proper expansion, in which the project name (tensorflow) is absent or "mixed" with the organization name (org.tensorflow) in the packages.
Logically, I see the project structure at a high-level more something like this:

tensorflow

tensorflow-core  (org.tensorflow.tensorflow.core.*)
tensorflow-framework (org.tensorflow.tensorflow.framework.*)
tensorflow-keras (org.tensorflow.tensorflow.keras.*)

ndarray (org.tensorflow.ndarray.*)

Adding the second tensorflow suddenly all make sense. But we it was not present in the initial code and we did not added it. So I don't think the actual irregularities with the module and the package names are coming from my proposal for ndarray but more from our other existing artifacts.
Now, when you talk about adding submodules under ndarray, while I'm not foreseeing any, it is not something impossible and it is a fair point. In that case, yes, having a distinct project name for all the "ND-I/O" stuff could be a better fit. For example, if we would hypothetically call this project something like NumJ, we could replace ndarray in the previous structure by:

numj

numj-buffer (org.tensorflow.numj.buffer.*)
numj-ndarray (org.tensorflow.numj.ndarray.*)
numj-linalg? (org.tensorflow.numj.linalg.*)

On the other hand, if we end up having a single ndarray artifact that contains everything we need, then the current proposal is fine.

The module still gets published under the org.tensorflow group, and it still uses org.tensorflow as package name. I really don't see how just removing a tensorflow- prefix is supposed to clarify things.  Why not publish that code under your own name? Like com.github.karllessard.ndarray?

Why not publish that code under your own name? Like com.github.karllessard.ndarray?

Hehe, yes we discussed about that as well if you remember but we need to check how possible it is to move out this code now that it has been submitted in that repository. I agree that it is just a small change for now, still I think it can attract more potential users than preserving its actual name, tensorflow-tools.
I added this topic as part of the agenda for this Friday, I feel there is more to be discussed about this change as well.

TensorFlow is open source under Apache v2. We can fork it and do whatever we want with it, there's no issues at all. What would cause problems is if we wanted to move the code to an organization that requires to own the copyright on the code. That would be problematic, because everyone that contributes code to TensorFlow keeps their copyright. Though organizations like Eclipse these days don't require copyright transfer, so that's usually alright.
In any case, I think trying to distance ndarray from TensorFlow would be a mistake. As I mentioned at https://github.com/bytedeco/javacpp/issues/391#issuecomment-626085198, this library actually becomes useful to end users when it starts supporting GPU memory and things like that. And to support that, we need to code ops in native code, at which point, we might as well be using existing ops from the core of TensorFlow, something that BTW is going to look a lot like PyTorch: https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html
Because of that, regardless of how we name things here, in the end, if ndarray ends up supporting GPUs, TensorFlow for Java will become known as the NumPy of Java anyway, so why try to stop that?

I disagree that the library needs to support GPU memory to be useful to users. Most DL inference happens on CPUs at the moment, and decent multidimensional array support is a precondition for deep learning. Numpy doesn't support GPUs, I would say that's been pretty successful even before GPUs and deep learning took off.
The main thing we need is for everyone who is using Tensors on the JVM to agree on a common interface, the same way people in Python have agreed on numpy as the interface. Having hundreds of ndarray classes, all of which support a different set of operations makes it very difficult for people to switch between frameworks to exploit their various strengths. Plus the cognitive overhead for developers when they have to learn yet another way of doing indexing and other operations to learn a new library isn't helpful.

I disagree that the library needs to support GPU memory to be useful to users. Most DL inference happens on CPUs at the moment, and decent multidimensional array support is a precondition for deep learning. Numpy doesn't support GPUs, I would say that's been pretty successful even before GPUs and deep learning took off.

The reason the Python community came up with NumPy in the first place is because the language is a lot slower than C/C++. Fortran never needed anything like NumPy, and most of its users still don't use any similar library. It's (almost) the same for Java.

The main thing we need is for everyone who is using Tensors on the JVM to agree on a common interface, the same way people in Python have agreed on numpy as the interface. Having hundreds of ndarray classes, all of which support a different set of operations makes it very difficult for people to switch between frameworks to exploit their various strengths. Plus the cognitive overhead for developers when they have to learn yet another way of doing indexing and other operations to learn a new library isn't helpful.

Oh, sure, I fully agree. For example, even if most users use ndarray on their laptop CPUs, that's still useful, because their code will be able to run on server GPUs and TPUs. That's the whole point, in my opinion: Having the option of acceleration.

How about we create a little poll for this?

tensorflow-ndarray ?
ndarray?
something else?

I disagree that the library needs to support GPU memory to be useful to users. Most DL inference happens on CPUs at the moment, and decent multidimensional array support is a precondition for deep learning. Numpy doesn't support GPUs, I would say that's been pretty successful even before GPUs and deep learning took off.

The reason the Python community came up with NumPy in the first place is because the language is a lot slower than C/C++. Fortran never needed anything like NumPy, and most of its users still don't use any similar library. It's (almost) the same for Java.

Sure, except that pushes the cost of writing it onto the users of the library. We can write fairly fast linear algebra on CPU in Java, C2 and GraalVM EE have pretty good autovectorisation but it requires tight code to get the autovectorisation to trigger in the right way. Having a CPU tensor library that everyone agrees on is still beneficial, especially if it has NUMA support.

The main thing we need is for everyone who is using Tensors on the JVM to agree on a common interface, the same way people in Python have agreed on numpy as the interface. Having hundreds of ndarray classes, all of which support a different set of operations makes it very difficult for people to switch between frameworks to exploit their various strengths. Plus the cognitive overhead for developers when they have to learn yet another way of doing indexing and other operations to learn a new library isn't helpful.

Oh, sure, I fully agree. For example, even if most users use ndarray on their laptop CPUs, that's still useful, because their code will be able to run on server GPUs and TPUs. That's the whole point, in my opinion: Having the option of acceleration.

Agreed, but large CPUs in servers are also acceleration over laptops (especially with AVX-512 or Epyc 2s).

How about we create a little poll for this?

tensorflow-ndarray ?
ndarray?
something else?

So I was thinking that the best way to make sure that we won't rename this library later is to check how it could actually scale by adding a linear algebra layer. I want to do a few experiments first and I'll come back to you with a proposal.

Using GPUs? What are you proposing exactly?

Using GPUs? What are you proposing exactly?

No, I mean by adding basic linear algebra operations to this API, accelerated by TensorFlow when dealing with tensors.

If you're wondering about performance on CPU, try out this sample code:
https://github.com/bytedeco/javacpp-presets/tree/master/llvm/samples/polly
On my laptop, with usePollyParallel = true, I get this kind of result:
$ mvn clean compile exec:java -Djavacpp.platform.host
LLVM with Polly: 606.978191ms. c[0] = 504.840637
Now starting pure Java benchmark: This may take a minute.
Pure Java: 26966.741211ms. c[0] = 504.840637

So, Java is basically 45x times slower. We can optimize the Java code some more, but we can also optimize the native code some more, and that's what libraries like MKL do. Granted, this is only a single example of very large matrix multiplication, but it's a very important example. In any case, the point is, in general, Java code just doesn't cut it for that kind of computation.

I was curious about MKL's performance, so I updated the sample code with a comparison against it, and it turns out to be fully 12x faster than LLVM with Polly and OpenMP:
$ mvn clean compile exec:java -Djavacpp.platform.host
MKL: 48.986203ms. c[0] = 491.768799
LLVM with Polly: 611.060513ms. c[0] = 491.768677
Now starting pure Java benchmark: This may take a minute.
Pure Java: 27133.617122ms. c[0] = 491.768677

That makes it over 550x faster than unoptimized Java (and that's on my dinky little laptop, which can also probably pull off 10x that with CUDA). You can try as you might to optimize that code in Java, but there's no chance in hell you'll get closer than 10x from MKL, as per, for example, https://github.com/fommil/netlib-java#performance (which says that MKL is slower than Java for small matrices blah blah blah but Intel has already solved this, see https://software.intel.com/content/www/us/en/develop/articles/intel-math-kernel-library-improved-small-matrix-performance-using-just-in-time-jit-code.html), so it's really rather pointless to code that kind of thing in Java...

I think Karl is proposing to figure out what the API for linear algebra would look like on top of the ndarrays. It's not about picking a specific package or backend like MKL, but before embarking on doing it it would be good to know what the API shape is. We can always get a pointer out to the memory and hand it to MKL for the low level operations.
Also the pure Java matrix multiply you linked isn't a good test against MKL, it's not blocked, nor is it trying any of the standard tricks to accelerate matrix multiplies in any language, so it's not surprising it's very slow by comparison. Plus C2 and GraalVM EE can occasionally be very fussy about what they autovectorise and how they do it, so you need to contort the Java code a little to make the compilers happy to get the highest performance.
I have some hopes that the Vector API coming in Panama will allow us to get closer to MKL's performance in pure Java, but as in Java we don't have control over register allocation we're not likely to get all the way there.

I have some hopes that the Vector API coming in Panama will allow us to get closer to MKL's performance in pure Java, but as in Java we don't have control over register allocation we're not likely to get all the way there.

Which is exactly my point, why bother optimizing it in Java in the first place? There was actually a blog post about that, I was trying to find it back, but still can't find it anymore... I think it's this one, but it doesn't give numbers: https://richardstartin.github.io/posts/vectorised-algorithms-in-java

They just aren’t that different, and it’s easy to translate between the two. I wouldn’t expect it to be fast yet though. I have no idea what the scope of work involved in implementing all of the C2 intrinsics to make this possible is, but I assume it’s vast. The class jdk.incubator.vector.VectorIntrinsics seems to contain all of the intrinsics implemented so far, and it doesn’t contain every operation used in my array multiplication code. There is also the question of value types and vector box elimination. I will probably look at this again in the future when more of the JIT compiler work has been done, but I’m starting to get very excited about the possibility of much faster JVM based data processing.

That was 2 years ago, and basically nothing more since. I'm not saying it's impossible to do that kind of thing in Java, but no one is putting resources into it, and there's political issues with OpenJDK and/or Oracle not wanting to implement "fast math" in the JDK, which is not helping, and is something you could help with since you work at Oracle, so please try to do something about it. :)
Anyway, chip shops put a lot more money in things like CUDA and MKL, and they give them out for free, so you know, let's use that. MKL-DNN/DNNL/oneDNN or whatever it's named these days, is open source and Intel is even allowing contributions from ARM now: https://github.com/oneapi-src/oneDNN/

oneDNN CPU engine is used to execute primitives on Intel Architecture Processors, 64-bit Arm Architecture (AArch64) processors, and compatible devices.

So, you know, why fight this? Java is fine to do prototyping and to implement custom one-shot algorithms, but for widely used algorithms like matrix multiplication, it's not useful.

Intel and Arm are both contributing heavily to the vector API. It's moving ahead and the performance issues are being fixed (there are plenty of commits each week as you could see by checking the panama mailing list). It won't be all the way done till the JVM has inline types as they are necessary to remove all the boxes and give C2 a type wide enough to store the vectors in.
There are some issues with fast math in Java as the language spec doesn't allow things to be reordered if it changes the outcome. However that doesn't apply to the vector API because if you use it you've allowed the JVM to compile your operations in a way that improves perf.
I happen to quite like writing Java, and I'm excited to write more numerical code in it. I'm hopeful that the ability to fuse together operations by having them all in the same source language (plus allowing the compiler to inline across matrix multiplication and activation function boundaries) will make it possible to develop fast enough ml libraries without using native code. On the CPU anyway, the accelerator story is still further out.
Anyway, I'm not arguing against basing things on top of the mkl, in the short term that's the best choice for CPUs. Intel ones at least, the last time I checked it ran pretty badly on Epyc 2s, which is very disappointing given the theoretical performance of AMDs chips.

So after experimenting a few things on my side, a functional API at the tensor level (including linear algebra operations) would probably fit better in the Operand interface than the NdArray.
The main reason of this is that it will allow us to chain operations using TensorFlow without fetching unnecessary tensor data from intermediate results. Also, TensorFlow operates on constants, not tensors, meaning that we would also need to convert a tensor automatically to a constant before each function call if have this API at the NdArray level.
I won't get to deep on the details here (let me know if you are interested) but basically what this mean is that having a generic linear algebra API in the ndarray module could be interesting addition in the future but is not a priority in the scope of the TensorFlow project. We should first focus on simplicity and performance to leverage the best of TensorFlow rather than abstraction.
Still, ndarray is a better name for this library compared to tensorflow-tools, as it is  descriptive of what it contains or what should be included in it. And while I agree that tensorflow-ndarray makes the module naming more homogeneous within this repository, it would be misleading as the library is not build on top of TensorFlow itself, like the other modules are (I continue to think to my point that the inconsistency between the package and the module names comes from the other modules, not this one).
So I suggest that we merge this PR as proposed and move on other improvements (e.g. I see a possibility to work exclusively with the rich TType interfaces rather than going back and forth from/to Tensor).

I happen to quite like writing Java, and I'm excited to write more numerical code in it. I'm hopeful that the ability to fuse together operations by having them all in the same source language (plus allowing the compiler to inline across matrix multiplication and activation function boundaries) will make it possible to develop fast enough ml libraries without using native code. On the CPU anyway, the accelerator story is still further out.

I also happen to share this personal preference, but we have to be realistic...

Anyway, I'm not arguing against basing things on top of the mkl, in the short term that's the best choice for CPUs. Intel ones at least, the last time I checked it ran pretty badly on Epyc 2s, which is very disappointing given the theoretical performance of AMDs chips.

Right, OpenBLAS is a better candidate as foundation than MKL.
Things are getting fixed for AMD as well: https://github.com/xianyi/OpenBLAS/issues/2381
oneDNN is also using it for ARM instead of MKL: https://github.com/oneapi-src/oneDNN/commit/9728df553084f05664198616a243a80a56b188fd

Still, ndarray is a better name for this library compared to tensorflow-tools, as it is descriptive of what it contains or what should be included in it. And while I agree that tensorflow-ndarray makes the module naming more homogeneous within this repository, it would be misleading as the library is not build on top of TensorFlow itself, like the other modules are (I continue to think to my point that the inconsistency between the package and the module names comes from the other modules, not this one).

I'm OK with all your saying. However, I'm seriously skeptical that a module using the group and package org.tensorflow is going to achieve your goal here. Why not use something that doesn't contain "tensorflow" in it? Keeping "tensorflow" in there is severely misleading. Let's get this right so we don't have to rename it again.

However, I'm seriously skeptical that a module using the group and package org.tensorflow is going to achieve your goal here.

To give it another organization name, it needs to be moved out the TensorFlow org as well. Let me check with Joana what it is possible to do here.

Here's another idea. Swift is putting their Keras-like API in a separate repository https://github.com/tensorflow/swift-apis, but we could go the other way around instead putting stuff like ndarray in a repository like https://github.com/tensorflow/java-ndarray and clarifying in the README.md file that this is a 100% pure Java library that doesn't depend on the TensorFlow C/C++ runtime. I think that would put enough distance with the rest of the code, making it sufficiently clear, even if it remains in the org.tensorflow group.
BTW, TensorFlow.js https://github.com/tensorflow/tfjs has plenty of backends that don't depend on C/C++ and all their subdirectories start with "tfjs-"...

we could go the other way around instead putting stuff like ndarray in a repository like https://github.com/tensorflow/java-ndarray

I like that idea, actually. I've already requested Joana about moving the code out of the TensorFlow org but I will follow up with her with that suggestion as well. I guess it is a matter of how fast we can get this new repository ready.

Thinking about this some more, and from a strictly branding point of view, the fact that it's "TensorFlow" clearly isn't causing any issues for TensorFlow.js and its non-native backends, so I don't think this is the problem here either. It's more about emphasizing its state as "100% pure Java", so what about naming the module "tensorflow-java-ndarray"? That is like the "Java NdArray" project from the TensorFlow organization. I feel it contrasts enough with the other modules too:

tensorflow-java-ndarray
tensorflow-core-api
tensorflow-core-platform
tensorflow-framework
...

We could also have a tensorflow-java parent module containing tensorflow-java-ndarray and other potential pure Java modules...

Hey @saudet , sorry if I didn't come back to you sooner on this, I was waiting for some feedbacks from Joana before continuing this discussion (maybe to be discussed at the meeting on Friday, I won't be there though).
Personally, I think that the best of your suggestions is to have a separate repository under the TensorFlow organization to host this code, i.e. github.com/tensorflow/java-ndarray. If we all agree on this, then we can forget about all the other options and focus on this one. The good thing too would be that we could proceed to the renaming as suggested by this PR and then it is simply a matter of moving the code when the other repository is ready but the signature of the packages and module will remain the same (maybe at the exception that org.tensorflow:ndarray won't have org.tensorflow:tensorflow-java as a parent anymore, which is not a big deal). Wdyt?

From a branding perspective, putting it in another repository makes sense I think, but from a technical perspective, if the API still isn't stable, it will entail additional burden on the part of developers whenever the API changes.  Trust me, syncing multiple repositories manually all the time isn't pretty. You need to be sure about this.
BTW, I think what I like about "java-ndarray" is the fact that it can refer to a project that is named "Java NdArray at the TensorFlow organization". If we don't put "java" everywhere, it ends up being the "NdArray project at the TensorFlow organization", which doesn't have the same ring to it... In any case, as I mentioned many times already, I don't think that a 100% pure Java NumPy-like library is going to be useful. We need native backends...

if the API still isn't stable, it will entail additional burden on the part of developers whenever the API changes.

It might get things a bit more complex yes but I really see this project as something that can evolve at a different pace than the Tensorflow Java API itself so yes, make sense again to have it in a separate repository once we have it.

If we don't put "java" everywhere, it ends up being the "NdArray project at the TensorFlow organization", which doesn't have the same ring to it

Personally, I still prefer with the java, as pretty much all Maven artifacts are implicitly "java-something" (yeah I know, we do have an parent artifact called tensorflow-java but that is because tensorflow was already taken by 1.x). Also, the repository itself (java-ndarray) will take care of distinguishing it from other possible confusion with other languages supported by TensorFlow.
So based on this, and I think @Craigacp also agreed on this, I'll rebase my actual PR and merge it with this name, increasing the version to 0.2.0 to prevent bad surprises from users actually using our snapshots. Thanks a lot for all your advices.

I did some interesting discoveries with this PR, our actual configuration for running unit tests in tensorflow-framework does not work when we increase the Maven project version (like it is done here to 0.2.0-SNAPSHOT).
The reason is simple: we depend on the tensorflow-core-platform to load the native artifacts required for testing. The problem though is that when the version is increased, the artifacts for all platforms (mac, linux, windows) have not been deployed on Nexus, meaning that the only native artifact available is the one for the current platform.  The reason it was working before is because we have already deployed 0.1.0-SNAPSHOT on Nexus, from the previous build.
I tried to depend then on tensorflow-core-api:<platform> directly but I'm getting now an error for the MKL platforms that libiomp5 is not available, we need to include the javaCPP MKL dependency to download it. So the list dependencies to be added depends on the platform itself.
So as a temporary solution before we find something cleaner to fix this issue (maybe out-of-scope of this PR), I suggest to disable the tensorflow-framework unit tests when the platform extension is set (mkl, gpu, ...).

Build with mvn -Djavacpp.platform.host ..., it should work fine.

@saudet the problem is occurring in tensorflow-framework but all those javacpp.* properties are only processed under tensorflow-core

Maven profiles work with transitive dependencies, that's fine.

OK you are right, in fact I mixed up things a bit and realize I got this error only locally when I was not setting the javacpp.platform property. Setting it fixed the issue, I'll rollback my changes and try again on CI, thanks


minimum api

which minimum api does this custom component require？is it 14(ICE_CREAM_SANDWICH)?

@aaaking RTFM


Fix getTypeFacts for pattern template literal types

This PR fixes a stack overflow that occurs when calling getTypeFacts on a pattern template literal type (such as `foo${string}`). The PR also adds normalization of template literal types consisting of only ${string} placeholders. Such template literal types are now reduced to just string.
Fixes #41651.

@RyanCavanaugh @DanielRosenwasser I think we should backport this. Currently getTypeFacts causes a stack overflow when invoked on a pattern template literal type (or a union containing a pattern template literal type). This for example happens with the ||, &&, ??, and postfix ! operators.

Will this be released before 4.3? (ex: in 4.2.4)

It should be fixed in all releases since 4.2.

@ahejlsberg Ok, so it's another issue: https://twitter.com/zoontek/status/1376606905729232897 (I use template string inference and recursive types - it doesn't happen on 4.3 dev version, so I will track commits to check what fixes it).
Sorry for the bothering!


[Feature Request]: Implementing Levenshtein Distance Algorithm in C++ under DSA folder.

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
I propose adding the implementation of the Levenshtein Distance Algorithm in C++ to the DSA (Data Structures and Algorithms) folder. The Levenshtein Distance is crucial for measuring the difference between two sequences, making it invaluable for applications like spell checking, DNA sequencing, and natural language processing.
Use Case
This feature would enhance my use of the project by providing a robust algorithm for calculating edit distances between strings efficiently. For instance, in scenarios where accurate similarity metrics are required, such as autocorrect functionalities in text editors or determining string similarity in search algorithms, the Levenshtein Distance Algorithm would significantly improve accuracy and performance.
Benefits
This feature would be beneficial is in a plagiarism detection tool that needs to compare multiple documents to identify similarities based on textual content. Implementing the Levenshtein Distance Algorithm would facilitate accurate comparison and detection of similarities or differences between documents.
Add ScreenShots
No response
Priority
High
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I have starred the repository

@Ajay-Dhangar sir, kindly assign me this issue under GSSOC'24.


Interaction

I would like to think about how to extend the underlying grammar to include support for various forms of interaction.
Some sample interactions:

Clicking on an interval representing a group of data to see the subdivision of smaller data points.
Highlighting related data points when hovering on one of them.
Zooming in or out on specific regions of large, data-rich graphs.
Hiding/displaying individual sets of data.

The basic goal is to make computerized graphics take full advantage of the screen medium and allow viewers to perform analytical work not necessarily envisioned by the designer of the original graphic.
I know a number of people are currently working in this area. I plan to post research notes to this issue as I find interesting things to read.

A filter Interaction is nicely realized by http://square.github.io/crossfilter/


ubuntu 16.04 is nearing EOL

ubuntu 16.04 EOL date is fast approaching (30 april 2021) as per #232 we can start announcing the deprecation of this version

this process has been picked up in the community and has been underway for a while, so closing


Fix (another) typo in InputDecorationTheme.debugFillProperties()

This issue was reported and fixed by Maido Käära in  https://github.com/flutter/flutter/issues/20374 and https://github.com/flutter/flutter/pull/20424
This version of his PR just adds a test.
Fixes https://github.com/flutter/flutter/issues/20374.


docs: added example search query

Codecov Report

Merging #3 (08b1205) into main (5c0f950) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main       #3      +/-   ##
==========================================
- Coverage   98.64%   98.61%   -0.04%
==========================================
  Files           3        3
  Lines          74       72       -2
==========================================
- Hits           73       71       -2
  Misses          1        1

Impacted Files
Coverage Δ

src/ghgql/__init__.py
100.00% <ø> (ø)

src/ghgql/ghgql.py
100.00% <ø> (ø)

src/ghgql/result.py
97.36% <100.00%> (-0.14%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


added new readme in gujarati

Proposed changes
I have translated the readme file to gujarati language
Types of changes
introduction of a new readme_guj.md file

@all-contributors add @anubhav1450 for content.


remove use of bind, as suggested in http://www.benjamincripps.com/pos…

…ts/react-no-bind

Haha, thanks for referencing my own blog.


Scroll sender window when sending from a file

If the user copies a text file into the code sender window that's larger than will fit in the window, then MKOB should scroll the sender window so the character that's currently being sent is always visible.

This apparently got fixed at some point, so I'm closing the issue.

I think the 'message' update methods probably fixed this.


Add support to build for PHP8.0

There are some major internal changes coming with PHP8.0 and causing some internal API changes and Macros being deprecated.
This ensures these cases are being taken care of.
More info:
https://github.com/php/php-src/blob/php-8.0.0alpha3/UPGRADING.INTERNALS#L47

Coverage increased (+0.03%) to 74.401% when pulling 952d02d8c96e0f1127963091a15db68c2e75568e on mmokhi:patch-1 into 5f76f838e5d44ca951a44bec6be1af05fae4bd93 on microsoft:master.

Thank you @mmokhi for your contribution! As I already explained, #1127 has taken care of this. Next time please submit pull requests to our dev branch, which should compile with php 8.0 alpha.

Closing this now, @mmokhi


branch.io not passing details after clicking link and installing app

We have a strange behaviour here with the branch-response after an installation of the app (currently testing only with iOS with the latest cordova plugin 2.6.18 and with custom domain)
First step is deleting the app. Then click on the link. Message to download is coming and working. After opening the installed app there is only a 50% chance of getting a
["+clicked_branch_link"]:true
It is not important to count this as a new install, it is only important for us to direct the user to the correct view in the app after an installation.
If the app is already installed, I can't remember any problems at all. This way is working fine.
Now you have this cool liveview on your page and when it isn't working, the last column (session referring link url) in the liveview is "null", if it works, the url is correctly filled (see attachment)

In the Details from the Live-View i can see these IDs:
Branch Identity ID:<PHONE_NUMBER>86355179
Branch Session ID:<PHONE_NUMBER>98977789
ID:<PHONE_NUMBER>28587572

And in the Plugin-Debug-Output those same ids are also in the response:
BNCServerInterface.m(747) Debug: Server returned: Status: 200; Data: {
    data = "{\"+clicked_branch_link\":false,\"+is_first_session\":true}";
    "device_fingerprint_id" =<PHONE_NUMBER>53151301;
    "identity_id" =<PHONE_NUMBER>86355179;
    link = "https://xxxx.xxxx/a/key_live_fim734jZcHMRnkUIquFBpckaqrefifPp?%24identity_id=466886649486355179";
    "session_id" =<PHONE_NUMBER>98977789;
}

We also tried getLatestReferringParams to get those data, but without more success.

Hello!
Are you still having problems? I know you received feedback through our support, but I wanted to make sure you got everything working.
Thanks,

I also saw similar issues where clicked_branch_link was not being triggered most of the time for new installs. Branch support had advised me to use webhooks instead. I haven't tried it yet.
What's the most bulletproof method?

Hey, I'm having the same issue, when I open the link (in Safari) and I install the app, sometimes the Deep Link doesn't work at all, this is what the logs looks like at that moment:
BNCNetworkService.m(219) Debug: Network finish operation https://api.branch.io/v1/install 0.639s. Status 200 error (null).
{"session_id":"516029442108308059","identity_id":"516029442087709188","link":"https://mel2.test-app.link?%24identity_id=516029442087709188","data":"{\"+clicked_branch_link\":false,\"+is_first_session\":true}","device_fingerprint_id":"516029441995093675"}.
2018-04-20 19:07:13.041211-0500 New Stand[31130:4815900] [branch.io] BNCServerInterface.m(771) Debug: Server returned: Status: 200; Data: {
    data = "{\"+clicked_branch_link\":false,\"+is_first_session\":true}";
    "device_fingerprint_id" =<PHONE_NUMBER>95093675;
    "identity_id" =<PHONE_NUMBER>87709188;
    link = "https://mel2.test-app.link?%24identity_id=516029442087709188";
    "session_id" =<PHONE_NUMBER>08308059;

It's weird because sometimes it works, this is so frustrating.
I'm using the branch-cordova-sdk (v2.6.24).


Fixed duplicate input id issue in clarifai vectorstore

Description:
This PR fixes the issue faces with duplicate input id in Clarifai vectorstore class when ingesting documents into the vectorstore more than the batch size.

@efriis
can I get a quick review on this ?
just made small changes to fix it.


Upgrade PyTorch Lightning version to 1.7.2 from 1.7

Warning 1 mentioned in the README has been fixed in a later version of PyTorch Lightning (changelog). Upgrading the PL requirement to 1.7.2 removes the error which becomes very annoying while debugging/testing.

Thanks for the pull request, this bug was indeed very annoying!


Fixed CSS code for Mozilla Firefox browser

This commit fixes an issue with the website's view on Mozilla's Firefox browser.
Before:

After:

nice!


Windows 11 instead of emulator

Using Windows 11 instead of emulator
The emulator are hard to run on any platform (Linux/Windows/Mac) with low resource(RAM/CPU/Storage).
Since Windows has announced android support,
I would like to use it instead of an emulator.
This can become very important in near future to develop app.

Windows 11 doesn't even support APKs right now. You probably need to wait a few months to see what Microsoft brings. By then, Android Studio team might add native bindings and Flutter might make use of it.

Hi @apt-x4869
As mentioned above, I will add that it is too early to conclude what kind of Android support Windows 11 has, if we can run it in debug mode or not.  I am on Windows 11 right now, we can't try it as of now so let's wait, try it, and make a proposal if it is feasible to do this when It releases
Closing. If you disagree, please write in the comments and I will reopen it.
Thank you


Made fractal algorithms more flexible so that colors can be customized

There isn't a gradient control for use in SimpleEffectDialog yet, and the gradient type that was used is still undocumented, and is internal within Pinta.Effects. This is intentional, as we have to get it right before making it part of the public API.
... but until then we could add a combobox so that the user can choose among a few pre-defined color schemes. What do you think?
Here's with the original colors (which is now the default gradient):

Here's with the same gradient stops but different colors:

Here's with the original colors but different stops:

I haven't looked through the code in depth, but one idea could be to have the EffectData take a gradient parameter rather than an enum, so that the effect's API doesn't change if we later add support for a widget that can specify a custom gradient.
The SimpleEffectDialog could then handle how to create a gradient widget, e.g. just showing a preset combo box for now

I understand that, but in order to do that we would have to clearly define the public interface of the gradient type, but since this isn't done yet what if we use an enum, and we make the enum-typed property and the gradient type internal? This way, the effect will be customizable within Pinta, but third parties won't notice the change in API until we make the property (and gradient type) public

Would this prevent writing a unit test since the property is internal?
I'm not sure if this is a high enough priority that it needs to be squeezed in right away though - why not take the time to implement a gradient API that you're happy with making public?

I was thinking that the feature would increase the popularity of the project and maybe bring more contributors, but yeah, we could wait to get it right first.
I am almost happy with the methods in my gradient class; although it may need some refinements, like methods for adding new gradient stops and such, and adding some static creational methods so that the API doesn’t change even if some feature is added to the class. But what do you think about the public interface of the class as it is now?
As for the testing, I think we could do [assembly: InternalsVisibleTo ("Pinta.Effects.Test")] so that any piece of code can be tested even if it's not part of the public API.

👍
I'm away for the next couple weeks so won't have time to review this for the next little while, sorry!

Now I've implemented the most important of your suggestions. The default gradients go from 0 to 1 (and I added a Resized() method to allow us to adjust the gradient to the values we need), and there is no base ColorMapping class.

What (besides the InvLerp method) is missing?

What (besides the InvLerp method) is missing?

I think just that, and documenting any remaining public methods like Resized()
There also was a suggestion before of using standard library methods instead of the manual binary search - any thoughts on that?

There seems to be some bug that is making the tests fail, even though the fractals look visually OK. I'll be looking into it

The problem was a comparison that was supposed to be greater or equal, but was only greater. I think it's ready for merging now.

Thank you! I think the changes look good 👍
After taking a look over the unit tests, one final question is whether GetRandomColor() is necessary or if we can just use some arbitrary fixed colours like black or white. Having random values in a test makes me nervous in general :)

It's understandable (and reasonable) that you want the tests to be deterministic and reproducible.
GetRandomColor() is not necessary as such, but it's more of a way to emphasize that a certain result holds no matter what color is chosen. Of course, it need not be non-deterministic; we can find some other way to test that.
In any case, I've now made the changes.

Looks good! Thank you for working on this


Support dinosr inference (pseudo w2v2 approach)

What does this PR do? Please describe:
The PR provides an adhoc way to support inference of the DinoSR tokenizer pre-trained in fairseq.  The idea is to reuse the existing wav2vec2 architecture in fairseq2 but load it with dinosr weights.  This approach is feasible because besides the pretraining-specific modules in wav2vec2 (inclduing mask_emb, quantizer, final_proj, and project_q) and dinosr (mask_emb, codebook_* and ema), which will not be used in fine-tuning or inference, both models are essentially identical as a stack of Transformer layers.

I think seamless_next would be better to host this model declaration. I have been very busy with the launcher work (finally finished). Let me revise the AWS installer script tomorrow and merge the initial seamless_next PR. As a quick follow-up we can move this to seamless_next then.

Sounds good, I kind of agree seamless_next would be a better place to host this model declaration.  Let me know when the initial seamless_next PR is merged.  Thanks!