Build error: Skipping static resource ... - max size is 2.1 MB

Hi all, I got problem when run yarn build.
⇒  yarn build
yarn build v0.24.5
$ react-scripts build
Creating an optimized production build...
Skipping static resource ".../react-example/build/static/js/main.aa0e5d44.js" (2.24 MB) - max size is 2.1 MB
Failed to compile.

static/js/main.aa0e5d44.js from UglifyJs
Unexpected token: operator (>) [./~/get-stream/buffer-stream.js:4,0][static/js/main.aa0e5d44.js:35510,23]

error Command failed with exit code 1.

I'm using the lates yarn with node 6.10.3
I created an example repo with build error here.
Can you help me to figure out why the project failed to compile? Thank you!!!

I think I found out the root of this error. CRA build failed if I use package got to do http request.
But I not sure this is the CRA's bug or not.

Replace got with request-promise solve the problem. Maybe there is a bug in get-stream used by got.

As mentioned in #321, how can I make UglifyJS supports minifying ES2015 code?

You can't. Please ask your dependencies to be compiled to ES5. Builds are pretty slow already, and compiling them with Babel is just going to make them slower. Additionally, Babel sometimes breaks code so we can't arbitrarily compile everything with it.

I see now. Thank you, Dan. I will close this ticket.

The message about "skipping" is misleading though. We should remove that message as it obscures the real issue.

Can someone explain what the cause and solution to this is? I'm getting the same when trying to build a pretty much empty create-react-app project with just a few dependencies added:
{
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4",
    "react-router-dom": "^4.1.1",
    "semantic-ui-css": "^2.2.10",
    "semantic-ui-react": "^0.68.4"
  },
  "devDependencies": {
    "@types/jest": "^19.2.3",
    "@types/node": "^7.0.22",
    "@types/react": "^15.0.25",
    "@types/react-dom": "^15.5.0",
    "@types/react-router-dom": "^4.0.4",
    "react-scripts-ts": "2.2.0"
  },
  "scripts": {
    "start": "NODE_PATH=src react-scripts-ts start",
    "build": "NODE_PATH=src react-scripts-ts build",
    "test": "NODE_PATH=src react-scripts-ts test --env=jsdom",
    "eject": "react-scripts-ts eject"
  }
}

Do I need to eject and modify something in Webpack to get it working?

@codeaid Please report this to react-scripts-ts, we are not affiliated with this package in any way.
That said, the important error is here:
static/js/main.2a54bce8.js from UglifyJs
Unexpected token: punc ()) [./~/source-map-loader!./src/routes/config.js:2,0][static/js/main.2a54bce8.js:60936,86]

What is /src/routes/config.js? Is this your code? Uglify appears to choke on that.

@gaearon - my config.js looks like this:
import asyncComponent from 'components/AsyncComponent';
const HomeScreen = asyncComponent(() => import('screens/HomeScreen'));
const LoginScreen = asyncComponent(() => import('screens/LoginScreen'));

export default [
    {
        path: '/',
        title: 'Home',
        component: HomeScreen,
        layout: true,
    },
    {
        path: '/login',
        title: 'Login',
        component: LoginScreen,
    },
];

The async loader is taken from here and looks like this.

There are similar issues because the underlying problem is the same: Uglify can only handle ES5 code. So if you give it ES6 it will choke. People having similar issues experience this because they are consuming dependencies that aren't compiled to ES5. The solution in this case is to raise an issue with authors of these packages and ask them to compile to ES5.
But in your case react-scripts-ts could be the issue. It probably doesn't fully support the import syntax and leaves something that isn't ES5 in the output, causing Uglify to crash.
The bottom line is:

If it's our issue please prepare a project with vanilla react-scripts reproducing it.
Otherwise it is likely a react-scripts-ts issue.

Also "it just adds support for TypeScript" is not very accurate description in my opinion. It is a fork, and forks can contain arbitrary mistakes.

Hmm, converted my routes to load like this:
import HomeScreen from 'screens/HomeScreen';
import LoginScreen from 'screens/LoginScreen';

...and it built successfully, so you're probably right. Looks like TypeScript doesn't support import syntax yet and chokes on that based on this issue.

Not to do with this specific PR, but same Uglify error.*
My IDE's syntax highlighting** was indicating an unresolved export in my component, so thinking I had done something wrong tried to import from within an NPM published module.
Bad idea, it turned out I managed to inadvertently import an ES6 file and determining why builds failed in the above-described manner was pretty tricky.
e.g.
MyComponent.js:
import {Action} from 'somecomponent/src';
node_modules/SomeComponent/
  /src          < ES6 the component is written in
  /lib          < ES5 that gets built and distributed
  /index.js     < where Action should have been exported from

* Posting in hopes this saves someone else (or future me) time resolving!
(Mods, if this isn't the right place apologies, please feel free to do whatever you see fit.)
** JetBrains (latest at time of writing).

I suggest you not to rely on IDEs for imports too much. They don’t really understand what’s going on, and can offer confusing suggestions. You should write imports the way they’re written in package documentation, regardless of whether IDE thinks it’s right or not.

Closing, as misleading message was removed in https://github.com/facebookincubator/create-react-app/pull/2645.
If you find this issue please check for Uglify error below: most likely that's the one you want to look at.

Dan you make an excellent point regarding IDEs.
If you happen to have/know a .vimrc that's ES6 ready, I'd be quite interested.
Either way thanks for your efforts! :100:


Fix index.md code snippet

Code snippet wasn't showing any code, also highlighting the wrong part of the code.

Internal previews

📄 File
🔗 Preview link

aspnetcore/razor-pages/index.md
Introduction to Razor Pages in ASP.NET Core

Thanks for the contribution.  Be sure to check out your name as a contributor about the 3rd of next month on the What's new page in the Community contributors section at the bottom.
Check out is:issue is:open label:"code help wanted" " and is:open is:issue label:"Good first issue"


SDK changes to support adapter

Background
SDK needs to check whether input asset in xcall is registered as an xERC20 and has a lockbox setup. If it does, then call the adapter to perform the lockbox deposit + xcall.

Next steps:

[ ] Deploy adapters to all other chains
[ ] Deploy registry to all other chains
[ ] Register alll current assets to all registries

Features has been added to SDK, Only needs to add config for adapter and registry, Once we register all the xTokens there


Last time entry in project

Hi there.
I'm trying to retrieve the latest time entry from a specific project.
The only way I think I could do it is with the User method my_time_entries but this method doesn't retrieve all User's time entries. Currently the method is retrieving my last 10 time entries, and none of them are from the project I want to use.
There is another method or another way to retrieve the last entry from a specific project??
I hope I've explained it well. Thanks :)

Hi @kanet77 do you think you could help me out with this??
Thanks :)

Hi @jvlobo. Sorry I didn't get back to you sooner.
I don't see a single API call to get the latest time entry for a given project. The best option I see is to filter time entries by time interval (say, working backwards one week at a time) until you see the project of interest.
I think the right answer is probably to use the Reports API which I haven't yet had the time to implement and test.

If you still have a need for this, TogglV8 now supports the Reports API. You might be able to simply iterate backward 30 days at a time (or some other interval) till you find one or more time entries.

Closing because this is more a limitation of Toggl.com than the togglv8 library.


add example support for floats and arrays

This PR adds support for the example keyword for floats and arrays. For arrays, it expects a format like:
MyArray []string `jsonschema:"example=A;B"`

@samlown I've added tests now

@samlown Sorry for not having gotten around to this until now, but I've now addressed your comments. I adjusted the Type == "" case to accept any kind of value instead of just JSON objects, and added a corresponding test.


How do I integrate this with publications?

How do I integrate this with publications? Would be helpful to see an example.

Hi @sys13, I will add an example for this


Add tooling support for indirect-error-match-pattern

Description:
Need tooling support for
type ER error<string, ErrorData>;

function testIndirectErrorMatchPattern() returns string {
    ER err1 = error("Error Code", message = "Msg");
    match err1 {
        ER(message = m, ... var rest) => return <string>m;
    }
    return "Default";
}

Reopening since language server related changes are yet to be done.

Closing the issue since the fixes have been addressed.


Internet explorer

The whole theme doesnt work on Internet explorer. Its a blank page
core ui admin template

please verify that you have the latest version

The demo is also broken on IE. I keep getting 404 errors when going to any page.

Fixed


primary button styles

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fix #227
Test URLs:

Before: https://main--sling--aemsites.aem.live/whatson/sports/olympics/olympics-schedule
After: https://227-cta--sling--aemsites.aem.live/whatson/sports/olympics/olympics-schedule

Looks good now.


Release Google.Cloud.AutoML.V1 version 2.6.0

Changes in this release:
No API surface changes; just dependency updates.

Created release for Google.Cloud.AutoML.V1-2.6.0


:package: Patch Redux + use RTK alpha for full ESM support

ℹ What is the context for these changes?

Addresses an issue with Redux Toolkit and ESM support. When attempting to use the package in Next.js an error would be present stating addListener was not exported. This was primarily due to the lack of support for ESM usage in Redux Toolkit, which version 2 aims to resolve.
This makes use of an alpha package and a patch in order to remove an EmptyObject type from the Redux CombinedState type.
After chatting with some of the team from Redux, it appears the only "downside" to removing EmptyObject from the CombinedState type is that we must ensure that our initialState passed to combineReducers has all keys defined (which we already did anyway).

State (w/ Demo)
Image

Before

After

🕹 Demonstration

...

🎩 How can this be tophatted?

...
✅ Checklist

[ ] Tested on mobile
[ ] Tested on multiple browsers
[ ] Tested for accessibility
[ ] Includes unit tests
[ ] Updated relevant documentation for the changes (if necessary)

/snapit


Incorporate user feedback

This includes:

Rotations
Corrected dates
Cropping issues
Corrected OCR

Probably all of this should go through some kind of manual screening. It makes sense to wait to see what the volume of data is like before building anything out.

I'm regularly pulling in rotations and OCR corrections.


Migrate documentation to senzing-garage

Pull requests:

https://github.com/senzing-garage/docker-compose-demo/pull/337


upgrade vert.x maven plugin version

https://github.com/reactiverse/vertx-maven-plugin/releases

Fixed in master.


sct_straighten_spinalcord: bad concatenation of warping fields for straight2curve

The display of the warping field is bad -problem at edges-.

This is due to a bad command when concatenating rigid and non-linear transfo:
isct_ComposeMultiTransform 3 tmp.straight2curve.nii.gz -R data_RPI_crop_normalized.nii.gz -i tmp.curve2straight_rigid.txt tmp.warp_straight2curve.nii.gz
If you invert the command such as:
isct_ComposeMultiTransform 3 tmp.straight2curve.nii.gz -R data_RPI_crop_normalized.nii.gz tmp.warp_straight2curve.nii.gz -i tmp.curve2straight_rigid.txt
here is the result for the display of the warping field:

N.B.: results are the same for the image afterwards

see issue 414 for same problem

I agree with that and I changed the order of transformation. This rule should be checked: should the affine transfo always be at the end of the list or just not at the beginning?


Install for any package fails on Cygwin

GVM will not install packages, it always reports that the archive is corrupted:
adamn@Eris ~
$ gvm version
==== BROADCAST =============================================

ONLINE MODE RE-ENABLED! All functionality now restored.

============================================================
Groovy enVironment Manager 1.3.5

adamn@Eris ~
$ gvm install grails

Downloading: grails 2.3.0

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  120M  100  120M    0     0  1194k      0  0:01:43  0:01:43 --:--:-- 1233k

Stop! The archive was corrupt and has been removed! Please try installing again.

I can also confirm that zip fails and unzips works.


Useless scrollbar in get-in-touch form

When opening the get-in-touch form a scrollbar appears, because the element is to big.

Assign this to me
Ty


colSpan and rowSpan missing in types for Table.Cell and Table.HeaderCell

Typescript is missing the prop colSpan of type number, and rowSpan of type number in <Table.Cell /> and <Table.HeaderCell />. They work as intended, just aren't typed.

I appears you don't type html props specifically. Closing for now, as this is probably intended.


Master tests failing

Master CI tests failing because the new password_resets_spec is trying to use a selenium driver for Capybara: https://ci.cartodb.net/user/javitonino/my-views/view/Builder/job/CartoDB-master-testing/1658/console
This only happens when trying to run multiple specs at the same time, in particular:
bundle exec rspec spec/requests/password_resets_spec.rb spec/requests/carto/oauth_provider_controller_spec.rb fails, but running any of those tests separately works.
Possible solutions:

Move the new spec to another spec group (this is the quick and dirty way)
Investigate why this is happening (maybe due to the include from acceptance_helper and some hidden assumptions about Capybara state) and fix it.

Just launched a manual run of the tests as acceptance: https://ci.cartodb.net/user/javitonino/my-views/view/Builder/job/CartoDB-master-testing/1665/

all :green_apple:


Add Replica::pending_tasks and pending_task_data

We already have Replica::all_tasks and all_task_data, so it may make sense to add Replica::pending_tasks and pending_task_data which could then join on the working_set table (as opposed to taking an arbitrary list of UUIDs).
I suspect the implementation of this would be relatively straightforward, and based on existing examples at both the Replica and underlying TaskDB levels in TaskChampion. The connection to C++ is now much simpler than it used to be, using cxx, so that should also be relatively straightforward. And then calling the result from TDB2 is also pretty easy. The tricky bit is implementing it in TaskChampion and then developing against that in Taskwarrior, since they are different repos -- but I can help with that. This is definitely worth doing even if it's not the biggest contributor to poor performance. Do you want to give it a shot? If so, let's open a new issue to track the work.
Originally posted by @djmitche in https://github.com/GothenburgBitFactory/taskwarrior/issues/3329#issuecomment-2335007219

Hey @djmitche,
I've started looking into this and have written a simple implementation of pending_tasks with the signature pub fn pending_tasks(&mut self) -> Result<Vec<Task>> . This was to match this part of the cpp function: https://github.com/GothenburgBitFactory/taskwarrior/blob/40ea3f2f5418ab5795e2aa0e73a1206c1d18cc1e/src/TDB2.cpp#L355-L367
I'm not entirely sure this is the way you'd prefer it was done, so I'd like to open a draft so you can have a look before I start on the pending_task_data function if that's alright with you?

Sure, that sounds great!

Hi @djmitche, I am planning to incorporate taskchampion for my tasks app. I was reading through taskchampion's docs and source code when I noticed the lack of a function which returns tasks based on an applied filter. I skimmed through taskwarrior's source to see how they implemented it and I believe they iterate through all tasks and check if each task satify the criteria rather than running an sqlite query.
Is this the intended way to implement filtering, or like Replica::pending_tasks, are they planned to be added in a future release?

The intended way is to select either Replica::pending_tasks or Replica::all_tasks and then filter from there. The idea is that especially the set of pending tasks is usually quite small, so it's easiest to just iterate over all tasks and match the required conditions, rather than find a way to represent all conditions in some programmatic way to be evaluated by the storage backend.

@Scandiravian did you want to make the relevant changes in Taskwarrior, as well (basically making these new Replica methods available in C++ and calling them in TDB2::pending_tasks)? That will require a little bit of coordination, since we will need to release a new version of Taskchampion first. And, I have things in a bit of an unstable state with #373, so that may take a little bit.

@Scandiravian did you want to make the relevant changes in Taskwarrior, as well (basically making these new Replica methods available in C++ and calling them in TDB2::pending_tasks)? That will require a little bit of coordination, since we will need to release a new version of Taskchampion first. And, I have things in a bit of an unstable state with #373, so that may take a little bit.

@djmitche I haven't coded C++ in more than a decade, so it depends on the complexity of the change. If it's simple, such as replacing method calls, I can probably figure it out; I'm happy to give it a go at least

When I started working on Chromium almost 2 years ago, I felt the same way -- I hadn't done C++ since 2000! I'm sad to report a decade hasn't done it any favors..
That timing sounds fine. Maybe I can finish up #373 by then!

@Scandiravian were you going to have a chance to work on this?

Oh, sorry, #3632 is the work on that - we can follow up there.


y value not correct ???

In t1 experiment the Y value is always positive ?

@lpatiny
I close #709,#701 can you check if y value is correct.


Add Data Worker

Create the Data Worker process to perform the following tasks:

Flush and re-Initialize the data cache
Populate the entire data cache

This only makes sense if the data cache is out of process, like in a Redis service. This demo will only have a single in-process cache implementation, so this functionality - although required in a real app - will not be addressed here.


Index out of Range after scanning a barcode, then putting my finger to block the torch

I scanned a barcode on a bag of Cheetos,<PHONE_NUMBER>64, and the dialog popped up as expected. I confirmed, and the camera was still running as expected. I put my finger over the torch, and got an exception.
If I do not scan the barcode first, I do not get an exception.
Any help would be appreciated!
Packages installed:
BarcodeScanner.XF (5.0.9)
NETStandardLibrary (2.0.3)
Xamarin.Essentials (1.6.1)
Xamarin.Forms (<IP_ADDRESS>2)
`[ScrollView] initGoToTop
[InputTransport] Input channel constructed: fd=91
[ViewRootImpl@8cef3a6[Result]] setView = DecorView@d3d34a0[Result] TM=true MM=false
[ViewRootImpl@8cef3a6[Result]] dispatchAttachedToWindow
[ViewRootImpl@8cef3a6[Result]] Relayout returned: old=[0,63][1080,2094] new=[28,796][1052,1360] result=0x7 surface={valid=true<PHONE_NUMBER>32} changed=true
[OpenGLRenderer] eglCreateWindowSurface = 0x706691a580, 0x7063bd2010
[ScrollView]  onsize change changed
[ViewRootImpl@8cef3a6[Result]] MSG_WINDOW_FOCUS_CHANGED 1 1
[ViewRootImpl@8cef3a6[Result]] MSG_RESIZED: frame=Rect(28, 796 - 1052, 1360) ci=Rect(0, 0 - 0, 0) vi=Rect(0, 0 - 0, 0) or=1
[ViewRootImpl@5e855a8[MainActivity]] MSG_WINDOW_FOCUS_CHANGED 0 1
[InputMethodManager] prepareNavigationBarInfo() DecorView@6a1abba[MainActivity]
[InputMethodManager] getNavigationBarColor() -855310
[ViewRootImpl@8cef3a6[Result]] ViewPostIme pointer 0
[ViewRootImpl@8cef3a6[Result]] ViewPostIme pointer 1
[OpenGLRenderer] eglDestroySurface = 0x706691a580, 0x7063bd2000
[OpenGLRenderer] endAllActiveAnimators on 0x7065b11700 (RippleDrawable) with handle 0x7089ddb420
[ViewRootImpl@8cef3a6[Result]] dispatchDetachedFromWindow
[ViewRootImpl@8cef3a6[Result]] Surface release. android.view.ViewRootImpl.doDie:7964 android.view.ViewRootImpl.die:7932 android.view.WindowManagerGlobal.removeViewLocked:497 android.view.WindowManagerGlobal.removeView:435 android.view.WindowManagerImpl.removeViewImmediate:124 android.app.Dialog.dismissDialog:518 android.app.Dialog.dismiss:501 androidx.appcompat.app.AppCompatDialog.dismiss:140
[InputTransport] Input channel destroyed: fd=91
System.ArgumentOutOfRangeException: 'Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index'
System.ArgumentOutOfRangeException: 'Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index'
`

I think this may be due to the Navigation.popModalAsync() in the sample code, since it's a flyout menu.

Yep that was it, closing.


What is the expected usage of source-built assets?

We need to decide what we expect users are going to do with the assets built from source-build.
For example, do we expect people to publish the NuGet packages that were built to some feed?  Do we expect consumers to restore these NuGet packages in their apps?
One usage I can see is for the RID-specific NuGet packages to get published.  For example, if someone wants to build a self-contained application that runs on Tizen, they would need to reference the runtime.tizen.4.0.0-armel.Microsoft.NETCore.App package.  But Microsoft doesn't build/publish that package.  So I could imagine a case where someone uses source-build to build a RID-specific Microsoft.NETCore.App package, and publishes it so users can build self-contained applications on platforms that aren't officially built by Microsoft.
@dotnet/source-build-contrib

As someone who intends to use the output of source-build for packaging .NET Core into some Linux distributions, I have some comments.
Our primary use case is that we want to replicate what Microsoft distributes as the downloadable ".NET Core SDK" using source-build and make that available to users of those Linux distributions.
These Linux distributions often target architectures that are not targetted by Microsoft. In addition to x86_64 (aka x64) and i686 (aka x86), we are interested in architectures like arm32hf, aarc64, or even ppc64le. So we want to be able to build/run .NET Core applications on these architectures too. I understand that many of these ports do not exist, so I am really talking about a hypothetical future.
As for packages and remote publising, we would like the behaviour to match upstream. I don't think we will be uploading to NuGet, but the spirit of open source demands the packages are built and that we have the ability to, in case we wanted to to. In other words, if NuGet.org was to disappear from the face of earth for a day, someone in the community could build .NET Core from source and upload it to a TemporaryNuGet.org.
For another example, it would be great if we could upload custom RID packages to a custom myget repository so everyone can start using .NET Core on these not-supported-by-Microsoft architectures or platforms too.

It would be nice if nuget packages that would be uploaded to nuget.org would be put together in a folder.
So in theory, we could reference that folder from a NuGet.Config file and offline (i.e. using only artifacts built by source-build)

build and run of an app that targets netcoreappx.y
build and consume libraries that target netstandardx.y.

Also, source-build should also the ASP.NET runtime store since that is also part of the Microsoft .NET Core distribution. We want to have the same sdk and runtime API surface.

Also, source-build should also the ASP.NET runtime store since that is also part of the Microsoft .NET Core distribution. We want to have the same sdk and runtime API surface.

This was implicit in my description, but thank you for making it explicit: .NET Core built by source-build should be identical (bit by bit, ideally), including the package cache, runtime store and anything else that is included in Microsoft's distribution of .NET Core. This is immensely important. Without this source-build will be considered a second class citizen and not the real .NET Core.

cc @gegenschall

It would be nice if nuget packages that would be uploaded to nuget.org would be put together in a folder. So in theory, we could reference that folder from a NuGet.Config file and offline (i.e. using only artifacts built by source-build)

Part of the trouble with doing that is we cannot do it fully offline. A lot of our library packages cross-compile for many different targets (i.e. older versions of netstandard, netcoreapp, other OS's) which we don't have all of the versions of the source to rebuild a full nuget package with all those targets without downloading other binaries (mostly older nuget packages) to be able to build successfully. So if the goal is to produce exactly what the .NET team produces then we will require being online and we won't be able to do it purely from the contents of a source tar-ball.
However if the goal is purely to build a new shared framework, CLI/SDK, ASP.NET store for a new RID then I think we can accomplish that offline.
So to add to the question how much of this do you guys expect to be able to build from source purely offline?

Part of the trouble with doing that is we cannot do it fully offline. A lot of our library packages cross-compile for many different targets (i.e. older versions of netstandard, netcoreapp, other OS's) which we don't have all of the versions of the source to rebuild a full nuget package with all those targets without downloading other binaries (mostly older nuget packages) to be able to build successfully.

Does that mean you are adding extra targets by patching existing nuget packages?

So if the goal is to produce exactly what the .NET team produces then we will require being online and we won't be able to do it purely from the contents of a source tar-ball.

Anything that is online can be cached off-line. The source tar bal-ball we have now is also packed with nuget packages.
We should build those things which are meaningful for the .NET Core version&RID.
So, when building .NET Core 2.1 (as an example).
I think the meaningful targets would be:

any nuget packages needed to build a netcoreapp2.1 (RID generic + RID built for)
any nuget packages needed to build a netstandard1.1 - whatever netstandard .NET Core 2.1 supports library (These are RID generic I assume)

Would it be good/feasible to generate and archive a tar file for each release?
That tar file should include the online dependencies cached + source-build.
We need to include the source-built commit id used to generate the tar file (if that isn't the case already).
I am talking a lot about these nuget packages. The first goal is to build the distributable (sdk/runtime) itself from source.
I find the nuget packages important because it brings full open-source during the build of an app/library too.
Please comment on how much sense this makes to you and how feasible it is.

Please don't take anything below personally, it's just a bunch of facts as they are without any decoration.

We need to decide what we expect users are going to do with the assets built from source-build.

The Primary goal is for The Packagers to be able to package .NET Core CORRECTLY for every operating system (and platform.)

The Packagers - People responsible for packages in specific OS, such as Linux distribution. These people know the right way to include packages in their distribution and will take care of updates as fast as possible.

Packaging correctly - Every Linux distribution has rules on what packages are allowed and what's a no-no. Most are quite strict and currently almost no Linux distribution can include .NET Core packages with clear consciousness. (Example of why we can't package netcore - it can't be built from source. We can not include pre-build binaries, we can not package dependencies with it!)

what the .NET team produces

...is not an Open Source product.

However if the goal is purely to build a new shared framework, CLI/SDK, ASP.NET store for a new RID then I think we can accomplish that offline.

Not for a new RID. For the "old" ones too. All our current packages are hacked together holding only by the power of will.

A question you may ask: Why package something we provide for download at https://www.microsoft.com/net/core ?

Because it should be a simple install dotnet command which pulls the correct dependencies, and later on it would take care of all the updates, everything. Another reason is that we are able to provide fixes in very fast pace, we are able to build for not yet released versions of Linux distributions. Microsoft doesn't do that. In the Linux world it's often about being on the edge, having the latest version, but Microsoft packages arrive half a year late. One whole major release late in case of Fedora.

@omajid and @tmds have made the technical points quite well, I'll pass on repeating those words.

Example of why we can't package netcore - it can't be built from source.

Is this still currently the case?  I was under the assumption that the current source-build code met the needs of building netcore from source.
(sorry if the below question has already been discussed at length, I'm relatively new to this effort)
One piece that is still confusing me is the definition of the requirement of being built from source.  Does this mean:

Every binary that is contained in the package needs to be built from source code - or -
Every binary that is contained in the package and all the tools used to build those binaries need to be built from source code.

I believe the definition we should be working with would be:

Using only the publicly available information and build tools (source, readmes, compiler, etc, but not arbitrary binaries), anyone should be able to reproduce the entirety of the built project. This means building all necessary binaries as well, and those produced binaries being identical to the official ones (outside of things like automated compiler optimizations).

I'm not sure if that actually clarified much though.

@eerhardt

Every binary that is contained in the package and all the tools used to build those binaries need to be built from source code.

Long story short, this would be the answer, @omajid could talk about it in more detail...
@aslicerh

I'm not sure if that actually clarified much though.

I think that it summed it up nicely =)

@weshaggard  wrote:

Part of the trouble with doing that is we cannot do it fully offline. A lot of our library packages cross-compile for many different targets (i.e. older versions of netstandard, netcoreapp, other OS's) which we don't have all of the versions of the source to rebuild a full nuget package with all those targets without downloading other binaries (mostly older nuget packages) to be able to build successfully. So if the goal is to produce exactly what the .NET team produces then we will require being online and we won't be able to do it purely from the contents of a source tar-ball.

This is an issue that we should sort out, then.
Is there any particular reason we couldn't handle older versions by building those older versions separately too? For example, could we not build netstandard (from source, using the output of source-build) and then provide that to a next build so we could build everything in source-build that needs netstandard to build?
I think the key here is that packages that were themselves built from source (using only .NET Core form source-build) may be available on disk; they shouldn't be fetched from nuget.org/myget.org.
I dont have a good solution for cross compilation (targetting other architectures or platform).

However if the goal is purely to build a new shared framework, CLI/SDK, ASP.NET store for a new RID then I think we can accomplish that offline.

Please consider this as a minimum goal. It will work, but doesn't put .NET Core in a good lit as a open source project if not every component can be built from source and patched as needed.

So to add to the question how much of this do you guys expect to be able to build from source purely offline?

Ideally, 100% of what is distributed by Microsoft from http://dot.net/core.
@4creators wrote:

From my perspective as user I would like to be able to patch source of any repo, build complete patched .NET Core SDK and use it as a main tool on machine it was built on including Windows.

I would like to emphasize building involves various binaries used for building - such as the buildtools used at build time.  "patch source any repo" should include patching these .NET Core-specific tools for building.
@tmds wrote:

Anything that is online can be cached off-line. The source tar bal-ball we have now is also packed with nuget packages.

That is true, but the tarball is not good enough for many Linux distributions. Those packages must be built from source too (recursively). It is a question of fixing bugs (can we fix issues in build tools? can we fix issues in libraries used at build time?) and security (if we find a CVE, can we fix it even if upstream doesnt want to?)
@eerhardt

Is this still currently the case? I was under the assumption that the current source-build code met the needs of building netcore from source.

Not really. Think of a packaging building process as a reproducible pipeline. For another language like Java, it looks like this:
OpenJDK sources + a prebuilt version of OpenJDK SDK + native compiler + native libraries -> new version of OpenJDK SDK
And the build systems used by Linux distribution track each component in the pipeline. There is a process for identifying every component used and to patch and rebuild it, locally, if upstream disappears, or the internet goes down worldwide. For example, if there is a bug in the native compiler, all we really need to do is fix that, and then rebuild OpenJDK.
For .NET Core and current source-build the process looks like:
.NET Core sources + prebuilt .NET Core SDK Version 1.1 + prebuilt .NET Core SDK Version 2.0 + nuget packages (that target various platforms) + nuget packages (that include various precompiled binaries) + nuget packages (for build tools) -> .NET Core 2.0 SDK
Unfortunately, there are a ton of components in this pipeline that are not currently built from source. We need to get to a state where, aside from one version of a .NET Core SDK that we can use for bootstrapping, every other component must be built from source and assembled in a way which makes sure what code was compiled using what tools to produce what binary.

One piece that is still confusing me is the definition of the requirement of being built from source. Does this mean:

Every binary that is contained in the package needs to be built from source code - or -
Every binary that is contained in the package and all the tools used to build those binaries need to be built from source code.

Ideally, the second one. Our ideal goal is for .NET Core to get to a state where users can get the sources/patch/build any component and still build a .NET Core with that. And be sure only those versions were used in the .NET Core build.
If it helps, I dont think all of this work needs to be done in source-build. We should try and get all the dependency nuget packages to a state where they can be built using source-build and then used to build a subsequent version of source-build.
@aslicerh

Using only the publicly available information and build tools (source, readmes, compiler, etc, but not arbitrary binaries), anyone should be able to reproduce the entirety of the built project. This means building all necessary binaries as well, and those produced binaries being identical to the official ones (outside of things like automated compiler optimizations).

Pretty much. There is a one time exception for bootstrapping. So we could use a prebuilt .NET Core SDK (and a small set of nuget packages) to start off the build process, but eventually every component used to build a final .NET Core SDK must have been built from source.

Based on the discussion, I see 3 levels:
Level 1: Capable of building the dotnet distributable with same .NET Core features as Microsoft
All binaries included in distributable are built from source
Level 2: Capable of building assets needed to build a netstandard library and netcoreapp
It is possible to build offline using source-build built artifacts only
Level 3: Capable of using the source-build output to build all binaries consumed by source-build
All binaries to build source-build are also built from source

That level 3 is the goal is stated on the readme:

Many Linux distributions have specific rules for official packages. The rules can be summarized as two main rules: source for everything, and consistent reproducability.
A key goal of this repository was to satisfy the official packaging rules of commonly used Linux distributions, such as Fedora and Debian.

The split-up in 3 levels is to mark significant 'open source milestones'.
Currently we are working towards level 1, for which we still have open issues like the 'runtime store' and the 'net46 omnisharp'.
@eerhardt is your question answered by the above replies?

@eerhardt is your question answered by the above replies?

I'm beginning to see the picture.  The part that still confuses me is what the expectations for the NuGet packages would be.  Do people expect to use the NuGet packages that were created by source-build in the same ways they use the NuGet packages that Microsoft produces?

Do people expect to use the NuGet packages that were created by source-build in the same ways they use the NuGet packages that Microsoft produces?

As a source-build user:
(Correct me if I'm wrong) I think we need these packages (level 2) to be able to do level 3.
As a dotnet developer:
I can develop a netcoreapp app/netstandard lib using these packages without access to NuGet.org.
So "source for everything, consistent reproducible" applies also to the application I develop using .NET Core.

I think we need to take a hard look at how we use NuGet and if there are alternatives. Speaking in the extreme, one question I ask is: "What would it mean to deliver the SDK such that NuGet packages aren't a part of it but all the experiences work?". If NuGet were a way to get "newer" bits or bits that are not part of the SDK only, would that be a better world for build from source.
I'm nervous about the idea of having multiple "correct" builds of the same version of the same package floating around.

It's not really the NuGet packages that are the problem.  It is the stuff that is contained inside the NuGet packages.  In order to build for netstandard1.x or net4x, you need to get those reference assemblies from somewhere.
And also, there are assets that are being source-built that aren't in the SDK.  For example, System.Data.SqlClient, System.DirectoryServices, System.Text.Encoding.CodePages, etc.  We would need to provide an alternative acquisition story for those packages if we didn't want to use NuGet.

I think we've said we want to make sure you can build the refs for any version of .NET Standard easily from source. Once that is possible, if we just provided a flat folder of reference assemblies as a fallback in the SDK, would that work?
Is this just for how we build the product or also that they need to be there for people using the SDK and targeting those platforms?

I think we've said we want to make sure you can build the refs for any version of .NET Standard easily from source. Once that is possible, if we just provided a flat folder of reference assemblies as a fallback in the SDK, would that work?

While we could build just a flat folder here and customize all our repo's to use it in this mode it would require extra work to fork builds which isn't ideal. I think if we want to do this we should build the packages like they already exist which would require less changes in our repos and would allow folks consuming an SDK offline to use them as well.

for any version of .NET Standard

What about net4x?  See https://github.com/dotnet/source-build/issues/125

Yup, I see that delivery of reference assemblies in general is the problem. We've started to use NuGet heavily for that but I'm wondering if there needs to be a simpler fallback that ships with the SDK that is just a flat folder of assemblies for a given platform. Maybe I'm suggesting things that really are "boiling the ocean" but as far as I can tell NuGet and build from source seem to be heavily at odds.

If we have (and can build) reference assemblies for all target frameworks, would we then be capable of building all .NET Core nuget packages? Or are we still missing something?
This means we'd have a build everywhere capability for all .NET platforms.
Without an understanding of the technical challenges/effort... I find this is the nicest solution.
If not feasible, we need to look at solutions for a restricted set of tfms (netcoreapp, netstandard, ?net46, ??).

This is not for how we build the product. This is for people using the product that was "built from source".

Supposing there was a built from source version that worked as the community wants, would there be any reason for Microsoft to not use that too? Are there any issues with this approach which would mean that Microsoft can not adapt it?

This is not for how we build the product. This is for people using the product that was "built from source".

I think this is for how we build the product (level 2 supporting level 3) and for people using it (level 2). I don't know if @eerhardt meant something else.

Supposing there was a built from source version that worked as the community wants, would there be any reason for Microsoft to not use that too? Are there any issues with this approach which would mean that Microsoft can not adapt it?

Currently, Microsoft's .NET Core build has/relies on artifacts which can't be built from source (on Linux). When we get to the point where everything can be built from source, I can't think up good reasons why Microsoft would want do it different.

If we have (and can build) reference assemblies for all target frameworks, would we then be capable of building all .NET Core nuget packages? Or are we still missing something?

That would get us a lot of the way there but not all the way as we still have to worry about back-compat with our nuget packages and as such we end up reshipping existing binaries from older nuget packages for some TFMs. For these we don't currently have live builds for them in the tip of our repo's so we would either need to bring back all live builds (and the #ifdef maintenance of those) or pull in more source from our older release branches to be able to build those. This is a cascading problem which might require us to do hoop multiple times.
I have tend to agree with @Petermarcu that our use of nuget packages to deliver the platform seems to be at odds with our build from source efforts here. Something will have to give, either we stop using nuget the way we are which requires an large architectural change for .NET Core or we compromise on being able to build exactly the same assets in build from source and our MS official builds.

Something will have to give, either we stop using nuget the way we are which requires an large architectural change for .NET Core or we compromise on being able to build exactly the same assets in build from source and our MS official builds.

I'm thinking about this a bit. I haven't come up with many different solutions. I guess one way of handling the build-everything-from-source requirement could be to split off the source-buildable parts of the NuGet packages. e.g. System.Data.SqlClient has a related oss.System.Data.SqlClient which contains the netstandard and netcoreapp parts. oss.System.Data.SqlClient is then the package that meets the rules. System.Data.SqlClient is the package maintained by Microsoft that doesn't need to meet the rules.

I guess one way of handling the build-everything-from-source requirement could be to split off the source-buildable parts of the NuGet packages. e.g. System.Data.SqlClient has a related oss.System.Data.SqlClient which contains the netstandard and netcoreapp parts. oss.System.Data.SqlClient is then the package that meets the rules. System.Data.SqlClient is the package maintained by Microsoft that doesn't need to meet the rules.

This could be handled similar to the way that some package managers handle repos: There could be a 'free' and 'non-free' set of NuGet repos. Maybe even a flag in NuGet that, if (not) set, would warn you if you are sure you want to use non-free packages. The marking of free/non-free could be at a package level or a repo level.

On the offline bit, is it so that .NET Core source builds still depends on some NuGet packages that Microsoft has no plans to open source? It's a shame that we cannot have 100% pure, transparent .NET Core source code after all the efforts by .NET and ASP.NET teams..

So to add to the question how much of this do you guys expect to be able to build from source purely offline?

100%, everything, no hanky-panky. Then we can create nice reusable layered docker and make our lives easier by ourselves. If there is a third party dependency, like ASP.NET-KestrelHttpServer depends on libuv, it is and should be open sourced. Then source-build will run the autoconf build script in libuv and place it in correct output dir. If there are some closed-source dependency, please list them down transparently and give a brief explanation on why they can't be open sourced or when they will be open sourced.
This, imo, will give the community great confidence in open-source mode of Microsoft without any doubts.

/cc @leecow Please create issues for actionable items from this discussion


cannot start job due to error: cannot scan the path \?azcopy

Which version of the AzCopy was used?
Note: The version is visible when running AzCopy without any argument
Which platform are you using? (ex: Windows, Mac, Linux)
What command did you run?
Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.
What problem was encountered?
How can we reproduce the problem in the simplest way?
Have you found a mitigation/solution?

At first glance, this looks like it might be some kind of syntax issue in your command line.    If so, can you compare their command lines to the failed one, and let us know of any significant differences?

I am kind of facing similar issue. But this is on Azure devops build task "AzureBlob file copy"
From the log I copied below
Uploading files from source path: 'd:\a\r1\a\dist\Website.Web\www\' to storage account: 'deveastpointwebsite' in container: '$web' with blob prefix: ''
##[command] & "AzCopy\AzCopy.exe" copy "d:\a\r1\a\dist\Website.Web\www\" "https://devpointwebsite.blob.core.windows.net/`$web"  --log-level=INFO --recursive

failed to perform copy command due to error: cannot start job due to error: cannot scan the path \\?\D:\a\r1\a\dist\Website.Web\www, please verify that it is a valid.

I am trying to copy the website assets into Azure storage static website container "$web" and it is a windows agent.

Usually that error means exactly what it says: that it can't access that particular disk path. Either because of a typo in the name, or some permissions issue.

Hi Im having the same problem.
I KNOW that "it can't access that particular disk path", but there is no typo in the name nor permission issue. The thing that is causing the pain is that the address is rewritten to \?\C:\MyFile.txt instead of C:\MyFile.txt which is where my file is

I have this issue, and I think the issue is with running the job on a Linux
agent. Have you tried poolImage: windows-latest?
On Wed, Jun 10, 2020 at 6:31 PM John Rusk [MSFT]<EMAIL_ADDRESS>wrote:

The \?\ prefix is harmless. It's just used by Windows to allow long
paths to be passed to all methods (beyond the MAX_PATH length limit). So we
add that prefix to everything. It's called an extended-length path. See
https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation
(For consistency, we use that prefix on everything, not just ones which we
know to be long).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-storage-azcopy/issues/960#issuecomment-642301587,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AE52M6ODCBLZALBCG2NF32DRWACUNANCNFSM4MHYYRDA
.

--
Michael Palumbo

If we want to copy a local file to storage account , below is the command
azcopy copy C:\localdirectory\ https://storageaccountname.blob.core.windows.net/container --recursive
Below command will not work (i.e. directly pointing to the file will get the above mentioned exception)
azcopy copy C:\localdirectory\test.txt https://storageaccountname.blob.core.windows.net/container --recursive

Hi,
I'm hitting the same error trying to deploy to a static web container.
@mpalumbo7 tried using windows-latest as the pool image.

 Is AzCopy being run from a script or manually through cli?
(There are no SAS or other secrets in logfile and it is safe to post it here)

Hi,
I'm hitting the same error trying to deploy to a static web container.
@mpalumbo7 tried using windows-latest as the pool image with no succcess.

What is the complete command you were using ?

Hi,
I'm hitting the same error trying to deploy to a static web container.
@mpalumbo7 tried using windows-latest as the pool image with no succcess.

What is the complete command you were using ?

This is the command the agent tries to run:
"AzCopy\AzCopy.exe" copy "d:\a\1\[folder]\*" "https://[storage_accountName].blob.core.windows.net/$web"  --log-level=INFO --recursive`

@Geims83, can you post the log file here? Is AzCopy being run from a script or manually through cli?
(There are no SAS or other secrets in logfile and it is safe to post it here)

Here is the log file:
2020-07-31T09:19:36.0521832Z ##[section]Starting: AzureFileCopy 2020-07-31T09:19:36.0886490Z ============================================================================== 2020-07-31T09:19:36.0887578Z Task         : Azure file copy 2020-07-31T09:19:36.0889338Z Description  : Copy files to Azure Blob Storage or virtual machines 2020-07-31T09:19:36.0890135Z Version      : 4.171.3 2020-07-31T09:19:36.0890970Z Author       : Microsoft Corporation 2020-07-31T09:19:36.0892166Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy 2020-07-31T09:19:36.0893674Z ============================================================================== 2020-07-31T09:19:39.4689432Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Accounts\1.9.1\Az.Accounts.psd1 -Global 2020-07-31T09:19:42.8216343Z ##[command]Clear-AzContext -Scope Process 2020-07-31T09:19:44.0879834Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue 2020-07-31T09:19:44.5707953Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope 2020-07-31T09:19:47.0345460Z ##[command] Set-AzContext -SubscriptionId ****************************** -TenantId *** 2020-07-31T09:19:48.4840033Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Resources\1.8.0\Az.Resources.psd1 -Global 2020-07-31T09:19:48.9303134Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Storage\1.9.0\Az.Storage.psd1 -Global 2020-07-31T09:19:49.8781450Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Compute\3.1.0\Az.Compute.psd1 -Global 2020-07-31T09:19:50.5685972Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Network\2.1.0\Az.Network.psd1 -Global 2020-07-31T09:20:00.8134981Z ##[command] & "AzCopy\AzCopy.exe" login --service-principal --application-id "***" --tenant-id="***" --aad-endpoint "https://login.windows.net/" 2020-07-31T09:20:00.8831143Z INFO: If you set an environment variable by using the command line, that variable will be readable in your command line history. Consider clearing variables that contain credentials from your command line history.  To keep variables from appearing in your history, you can use a script to prompt the user for their credentials, and to set the environment variable. 2020-07-31T09:20:01.2814577Z INFO: AzCopy.exe: A newer version 10.5.1 is available to download 2020-07-31T09:20:01.2816117Z  2020-07-31T09:20:01.9115037Z INFO: SPN Auth via secret succeeded. 2020-07-31T09:20:02.2338834Z INFO: AzCopy.exe: A newer version 10.5.1 is available to download 2020-07-31T09:20:02.2341200Z  2020-07-31T09:20:02.2487362Z Uploading files from source path: 'd:\a\1\xxxxxxxx\*' to storage account: 'abbuffetstaticstorage' in container: '$web' with blob prefix: '' 2020-07-31T09:20:02.2598451Z ##[command] & "AzCopy\AzCopy.exe" copy "d:\a\1\xxxxxxxx\*" "https://abbuffetstaticstorage.blob.core.windows.net/$web"  --log-level=INFO --recursive
2020-07-31T09:20:02.3012134Z INFO: Scanning...
2020-07-31T09:20:02.3013526Z INFO: Using OAuth token for authentication.
2020-07-31T09:20:02.6239020Z INFO: AzCopy.exe: A newer version 10.5.1 is available to download
2020-07-31T09:20:02.6242234Z
2020-07-31T09:20:05.0565524Z
2020-07-31T09:20:05.0567504Z failed to perform copy command due to error: cannot start job due to error: cannot scan the path \?\D:\a\1\xxxxxxxx, please verify that it is a valid.
2020-07-31T09:20:05.0568891Z
2020-07-31T09:20:05.1250051Z ##[command] & "AzCopy\AzCopy.exe" logout
2020-07-31T09:20:05.1740451Z INFO: Logout succeeded.
2020-07-31T09:20:05.5217283Z INFO: AzCopy.exe: A newer version 10.5.1 is available to download
2020-07-31T09:20:05.5219194Z
2020-07-31T09:20:05.6202805Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2020-07-31T09:20:05.8224404Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2020-07-31T09:20:06.6265116Z ##[error]Upload to container: '$web' in storage account: 'xxxxxxxx' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
2020-07-31T09:20:06.6970442Z ##[section]Finishing: AzureFileCopy`

Ok,
so it seems AITA.
I was having two problems. I was using the publish task at the end of the build job and I wasn't:

using a deploy task in the deploy job (so it seems the pipeline wasn't downloading the artifact from the build job)
I did not add the drop/ subfoder to my path (correct folder was `$(Pipeline.Workspace)/drop/whatever)

At the moment I'm still hitting a 403 error while trying to copy the files but this is definitely another issue.

Hi
I'm trying to archive in azure blob container from this path
azcopy copy "V:\Carreteras-CatVIS" "https://archivadohistorico.blob.core.windows.net/catvis?sv=2019-12-...." --recursive --block-blob-tier='archive' and this path is fine with permissions but always this error
failed to perform copy command due to error: cannot start job due to error: cannot scan the path \?\V:\Carreteras-CatVIS, please verify that it is a valid.
If I try to use the shared resource directly from the server "\srvdatos2012\CatVIS" I get this errors
2020/11/17 07:50:35 JobID=f93b58eb-9187-934a-7516-78a90323cd0d, credential type: Anonymous
INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /h
Any idea please?
regards

Hi
I'm trying to archive in azure blob container from this path
azcopy copy "V:\Carreteras-CatVIS" "https://archivadohistorico.blob.core.windows.net/catvis?sv=2019-12-...." --recursive --block-blob-tier='archive' and this path is fine with permissions but always this error
failed to perform copy command due to error: cannot start job due to error: cannot scan the path ?\V:\Carreteras-CatVIS, please verify that it is a valid.
If I try to use the shared resource directly from the server "\srvdatos2012\CatVIS" I get this errors
2020/11/17 07:50:35 JobID=f93b58eb-9187-934a-7516-78a90323cd0d, credential type: Anonymous
INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /h
Any idea please?
regards

Hey @kintela
Thanks for reaching out!
When you specify source using double quotes, you have to use \\ instead of \. This is because \  act as an escape sequence character when used inside the double quotes.
i.e. Instead of "V:\Carreteras-CatVIS" try "V:\Carreteras-CatVIS"

Sorry but when I use simple quotes I get the same errors I used double quotes because in the command used by Azure storage explorer i saw them there

Hi @kintela, have you figured out the problem?
I'm closing this issue as it appears to be about permission/syntax issues and not a problem with the tool itself.

I'm having the same issue when using azure DevOps deploy task
I can see the artifact is downloaded but can't find where it is downloaded to :(
I'm already on this for couple days now with no success :(

I am having the same issue.
failed to perform copy command due to error: cannot start job due to error: cannot scan the path \?\D:\a\r1\a\buildpath>\drop<pathtomyfile>, please verify that it is a valid.
Was there any resolution to this issue?

@kganesan1 could you please clarify which UI you are referring to?

Ignore my message. I was referring to UI from azure artifacts/Connect Feed/Nuget page. After posting the message, I realized that my build had a problem and did not really have the artifact and hence this error was legit.


Command donejs new for creating new projects

Create a new DoneJS CLI command donejs new for creating new things. The donejs add command should be kept and used for additive features to existing projects.
Currently, these commands are "creating": donejs add app|plugin|generator.

This would be a breaking feature and couldn't happen until 2.0.

Doesn't break if it is just added and add keeps it's original function which it should for the sake of all the intro docs (ours and others) that would need to be changed.

is maybe related also to this we should talk about that also https://github.com/donejs/donejs/issues/995
as the generator is able to create and detect existing projects we could let the api as it is

@andrejewski

Currently, these commands are "creating": donejs add app|plugin|generator.
This is not really true some of that can also be modifying existing projects.


[AutoPR cognitiveservices/data-plane/QnAMaker] Fix endpoint name clash

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/4548

(message created by the CI based on PR content)

Codecov Report

Merging #3862 into restapi_auto_cognitiveservices/data-plane/QnAMaker will increase coverage by 0.12%.
The diff coverage is n/a.

@@                                  Coverage Diff                                   @@
##           restapi_auto_cognitiveservices/data-plane/QnAMaker    #3862      +/-   ##
======================================================================================
+ Coverage                                               56.07%   56.19%   +0.12%
======================================================================================
  Files                                                    9414     9281     -133
  Lines                                                  190606   186484    -4122
======================================================================================
- Hits                                                   106877   104801    -2076
+ Misses                                                  83729    81683    -2046

Impacted Files
Coverage Δ

...gmt/datamigration/models/command_properties_py3.py
60% <0%> (-11.43%)
:arrow_down:

...re/mgmt/datamigration/models/command_properties.py
60% <0%> (-11.43%)
:arrow_down:

...rk/azure/mgmt/network/network_management_client.py
29.3% <0%> (-9.55%)
:arrow_down:

...els/network_configuration_diagnostic_result_py3.py
57.14% <0%> (-9.53%)
:arrow_down:

.../models/network_configuration_diagnostic_result.py
57.14% <0%> (-9.53%)
:arrow_down:

...ql/models/database_vulnerability_assessment_py3.py
50% <0%> (-7.15%)
:arrow_down:

...mt/sql/models/database_vulnerability_assessment.py
50% <0%> (-7.15%)
:arrow_down:

...mgmt-sql/azure/mgmt/sql/models/managed_instance.py
26.31% <0%> (-7.02%)
:arrow_down:

...-sql/azure/mgmt/sql/models/managed_instance_py3.py
26.31% <0%> (-7.02%)
:arrow_down:

...l/azure/mgmt/sql/models/managed_instance_update.py
26.31% <0%> (-7.02%)
:arrow_down:

... and 230 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ca7c32...11218a8. Read the comment docs.


Screen for leaderboards

Story #5 Leaderboards
Sub-Tasks:

[x] CSS
[x] Display top 5 players with their ELO
[x] Player-Search field
[x] Click on name to direct to profile
[x] Back to menu button

Estimate: 6h

25f503db2946cecc5dd45f68e54914f8ac553dd3


Setting pan_id not working

I tried to set the pan id for my USB stick, This does however not work.
The config expects the pan_id to be a string. The MQTT broker expects it to be a number.
Setting the pan_id to string "0x162a" literally saves it as a string and does not convert it to int 6754. Setting the pan_id to 6754 will save it as a string. With single quotes around it. That is not working either.

There is a similar problem with ext_pan_id: this one will simply be ignored & removed from the configuration.

I not the pan_id string issue also


Docker compose fails

Hi -- I'm trying to adapt a starter Python course that we wrote a few years ago for use with Binder to the template provided here since it provides a lot of really useful features in terms of self-testing of both code and comprehension.
We already make use of Docker so I was quite excited to think that I'd not have to go anywhere near npm since my previous excursions into the package manager have been... challenging. It might just be my poor understanding of node, but having forked the repository as a template and cloned it to my local machine I'm stuck on this:
 ⠿ Container code-camp-rebooted-gatsby-1  Create...                                        27.1s
Attaching to code-camp-rebooted-gatsby-1
code-camp-rebooted-gatsby-1  | /usr/local/lib/node_modules/gatsby-cli/node_modules/gatsby-telemetry/lib/telemetry.js:39
code-camp-rebooted-gatsby-1  |   store = new _eventStorage.EventStorage();
code-camp-rebooted-gatsby-1  |         ^
code-camp-rebooted-gatsby-1  |
code-camp-rebooted-gatsby-1  | SyntaxError: Unexpected token =
code-camp-rebooted-gatsby-1  |     at Module._compile (internal/modules/cjs/loader.js:723:23)
code-camp-rebooted-gatsby-1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
code-camp-rebooted-gatsby-1  |     at Module.load (internal/modules/cjs/loader.js:653:32)
code-camp-rebooted-gatsby-1  |     at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
code-camp-rebooted-gatsby-1  |     at Function.Module._load (internal/modules/cjs/loader.js:585:3)
code-camp-rebooted-gatsby-1  |     at Module.require (internal/modules/cjs/loader.js:692:17)
code-camp-rebooted-gatsby-1  |     at require (internal/modules/cjs/helpers.js:25:18)
code-camp-rebooted-gatsby-1  |     at Object.<anonymous> (/usr/local/lib/node_modules/gatsby-cli/node_modules/gatsby-telemetry/lib/index.js:21:18)
code-camp-rebooted-gatsby-1  |     at Module._compile (internal/modules/cjs/loader.js:778:30)
code-camp-rebooted-gatsby-1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
code-camp-rebooted-gatsby-1 exited with code 1

This is using the repo as cloned on Docker 4.4.2 (Intel Mac running Monterey 12.2.1) with no attempts at modification. Tracing this error via Google led me to this:

With what you provided I'd guess you're running an ancient version of Node:

.nvm/versions/node/v10.24.1
You'll need to use at least Node 14.15.0 with Gatsby

So it looks to me like something (in sharp?) is no longer compatible with what's being installed during docker-compose and hasn't had its version pinned to a compatible one (which seems strange given the sheer overwhelming detail of package-lock.json). Trying to manually install sharp as part of the build process hasn't worked for me. This is almost certainly a function of my lack of knowledge about node. Is there a pre-built image somewhere that I can use, or does that not work because the web site is not actually mounted from the local FS but baked into the build? It doesn't look like that's the case but...
Anyway, since then I've tried this tutorial, this tutorial, and this tutorial. The last one has gotten me closest to a running Docker image but installs someone else's package.json plan with various version locks that are only a bit more recent than this starter course. When I try to bring in the bits that seem to be missing (by comparing against your package.json file) I end up in npm ::hell::.
Is there a process/explanation for installing this stuff from scratch so that I'm not doing cargo cult stuff with npm, package.json, and package-lock.json?
If I can get this building I'd be happy to put a Docker image on the Hub so that a backup process exists as and when the build starts failing...
P.S. Do I also infer that each code cell currently launches its own Binder instance?

One more note... the docker/Dockerfile in space-course does still build but is also emitting warnings and refusing to run after running make build and make gatsby-dev:
 ERROR

Gatsby requires Node.js 14.15.0 or higher (you have v12.22.10).
Upgrade Node to the latest stable release: https://gatsby.dev/upgrading-node-js

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>dev: `gatsby develop --host=<IP_ADDRESS>`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR!


refactor: update user-management-module template

module update introduces local branch rules for main and development

closing, modules will be updated again if required.


feature/#1294_2 - new MapView.setMapCenterOffset method

The map center used to be projected into the screen center.
Now we have a possible offset from the screen center; default offset is [0, 0].
New class:

SampleMapCenterOffset: demo about the new MapView.setMapCenterOffset method - demo that can be reached at "More Samples / Events / Animate To with Map Center Offset"

Impacted classes:

MapView: created int members mMapCenterOffsetX and mMapCenterOffsetY with getters and a common setter; used it when constructing a new related Projection
Projection: created final int members mMapCenterOffsetX and mMapCenterOffsetY; added these parameters to constructors; impacted accordingly methods getScreenCenterX and getScreenCenterY; gently refactored
SampleFactory: added the new SampleMapCenterOffset demo
SampleMapSnapshot: impacted the use of a Projection constructor

When do you think this added feature will be available in the baseline?
On Wed, 5 Jun 2019, 17:24 monsieurtanuki<EMAIL_ADDRESS>wrote:

The map center used to be projected into the screen center.
Now we have a possible offset from the screen center; default offset is
[0, 0].
New class:

SampleMapCenterOffset: demo about the new MapView.setMapCenterOffset
method - demo that can be reached at "More Samples / Events / Animate To
with Map Center Offset"

Impacted classes:

MapView: created int members mMapCenterOffsetX and mMapCenterOffsetY
with getters and a common setter; used it when constructing a new related
Projection
Projection: created final int members mMapCenterOffsetX and
mMapCenterOffsetY; added these parameters to constructors; impacted
accordingly methods getScreenCenterX and getScreenCenterY; gently
refactored
SampleFactory: added the new SampleMapCenterOffset demo
SampleMapSnapshot: impacted the use of a Projection constructor

You can view, comment on, or merge this pull request online at:
https://github.com/osmdroid/osmdroid/pull/1353
Commit Summary

feature/#1294_2 - new MapView.setMapCenterOffset method

File Changes

M
OpenStreetMapViewer/src/main/java/org/osmdroid/samplefragments/SampleFactory.java
https://github.com/osmdroid/osmdroid/pull/1353/files#diff-0 (2)
M
OpenStreetMapViewer/src/main/java/org/osmdroid/samplefragments/data/SampleMapSnapshot.java
https://github.com/osmdroid/osmdroid/pull/1353/files#diff-1 (2)
A
OpenStreetMapViewer/src/main/java/org/osmdroid/samplefragments/events/SampleMapCenterOffset.java
https://github.com/osmdroid/osmdroid/pull/1353/files#diff-2 (135)
M osmdroid-android/src/main/java/org/osmdroid/views/MapView.java
https://github.com/osmdroid/osmdroid/pull/1353/files#diff-3 (33)
M osmdroid-android/src/main/java/org/osmdroid/views/Projection.java
https://github.com/osmdroid/osmdroid/pull/1353/files#diff-4 (39)

Patch Links:

https://github.com/osmdroid/osmdroid/pull/1353.patch
https://github.com/osmdroid/osmdroid/pull/1353.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/osmdroid/osmdroid/pull/1353?email_source=notifications&email_token=ALCUZICXKE2LGESVGVXG6GLPY7LDLA5CNFSM4HT3YH7KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GXZTILA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALCUZIAFLCJGHZP44COGRFLPY7LDLANCNFSM4HT3YH7A
.

ProjectionTest.java is missing offsets 0, 0

After I fixed the build, the change works great, I tested on emu and tablet with a compass-driven-orientation.  Haven't implemented yet switching between offset and no offset.

Oh, ProjectionTest, of course.
You didn't mention it before: what do you expect as a transition between different offsets? I guess something like should work:
// from (offsetX, offsetY) to (0, 0)
GeoPoint g = mapview.getCenter;
mapview.setMapCenterOffset(0, 0); // no invalidate
mapview.setExpectedCenter(g, -offsetX, -offsetY); // supposed a transparent transition
mapview.animateTo(g); // smooth animation

this looks good to me, just fix the merge conflict.


Unauthorized when I use an entity provider

First of all thank you for your bundle .
Now I'm stack with this problem : when I use the in_memory provider everything is working fine , but when I change the provider to use a Doctrine entity which implements UserInterface ,  I get 401 Unauthorized.
{
"code": 401,
"message": "Bad credentials"
}
This is my security.yml
security:
# encoders:
#     Symfony\Component\Security\Core\User\User: plaintext
encoders:
    Utb\MobileApiBundle\Entity\Abonne:
        algorithm: md5
# #     Symfony\Component\Security\Core\User\User:
# #         algorithm: md5

role_hierarchy:
    ROLE_ADMIN: ROLE_USER

# providers:
#     in_memory:
#         memory:
#             users:
#                 jeremie: { password: ace3i, roles: [ 'ROLE_USER' ] }
#                 admin: { password: password, roles: [ 'ROLE_ADMIN' ] }

providers:
    user_provider:
        entity:
            class: UtbMobileApiBundle:Abonne
            property: username

firewalls:
    dev:
        pattern:  ^/api-mobile/(_(profiler|wdt|doc))/
        security: false
        provider: user_provider

    login:
        pattern:  ^/api-mobile/login
        stateless: true
        anonymous: true
        provider: user_provider
        form_login:
            check_path: /api-mobile/login_check
            require_previous_session: false
            username_parameter: username
            password_parameter: password
            success_handler: lexik_jwt_authentication.handler.authentication_success
            failure_handler: lexik_jwt_authentication.handler.authentication_failure

    api:
        pattern:   ^/api-mobile
        stateless: true
        provider: user_provider
        lexik_jwt:
            authorization_header:
                enabled: true
                prefix:  Bearer
            query_parameter:
                enabled: true
                name:    bearer

access_control:
    - { path: ^/api-mobile/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
    - { path: ^/api-mobile, roles: IS_AUTHENTICATED_FULLY }

i don't know if i a missing anything

I have the same issue

dairdr, i resolved that issue by adding some parameters to my encoders
security:
encoders:
MobileApiBundle\Entity\Abonne:
algorithm: md5
encode-as-base64: false
iterations: 1

Now i have another problem, i can login correctly but all i get 401 error when i try to access my secured api routes. What can be the cause of that ?

letal1609, now it works, this is my config.yml:
To get started with security, check out the documentation:
http://symfony.com/doc/current/book/security.html
security:
encoders:
AppBundle\Entity\User:
algorithm: bcrypt
cost: 12
role_hierarchy:
    ROLE_ADMIN: [ROLE_USER, ROLE_SONATA_ADMIN]

providers:
    mysql_provider:
        entity:
            class: AppBundle:User
            property: username

firewalls:
    # disables authentication for assets and the profiler, adapt it according to your needs
    dev:
        pattern: ^/(_(profiler|wdt)|css|images|js)/
        security: false

    login:
        pattern:  ^/api/login
        stateless: true
        anonymous: true
        form_login:
            check_path:               /api/login_check
            success_handler:          lexik_jwt_authentication.handler.authentication_success
            failure_handler:          lexik_jwt_authentication.handler.authentication_failure
            require_previous_session: false

    api:
        pattern:   ^/api
        stateless: true
        lexik_jwt: ~

access_control:
    - { path: ^/api/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
    - { path: ^/api,       roles: IS_AUTHENTICATED_FULLY }

Everything it's working fine, i can get 401 and 200 HTTP status codes.

Thanks for sharing @dairdr. @letal1609 does it help ?

i have the same problem, when using plaintext encoder or memory provider there is no problem, when i switch to bcrypt 401 Bad credentials


Support osx and other apps, not just iOS

The platform restriction to iOS 8.0 appears to be arbitrary. OS X would benefit from SwiftDDP as well, as could TVos and WatchOS.

You're right, the platform restriction is arbitrary and is something I've been planning on looking at. The dependencies should all work on other platforms, as should the core code.


Enhanced whereConfig()

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

15 of 16 (93.75%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.1%) to 93.098%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Eloquent/Scopes/WhereConfig.php
15
16
93.75%

Totals

Change from base Build<PHONE_NUMBER>:
-0.1%

Covered Lines:
553

Relevant Lines:
594

💛  - Coveralls


As a CIF developer, I want the Funding Agreement form to be refactored with a custom commit handler, so I have more flexibility in the form layout.

Description:
Similarly to the milestone refaactor (#1007)
Acceptance Criteria:
Given I am a CIF developer
When I commit a funding agreement form
Then a custom form commit handler is called
And both funding_agreement and additional_funding_sources tables are populated
Given I am a CIF developer
When I add, remove or reorder fields on the funding agreement page
Then there is only a single form to change
Development Checklist:

[ ] there are two schemas, funding_parameter and funding_agreement. Probably we should just have one called funding_parameter (for consistency; usually schema matches table name)
[ ] write new, single json schema, using an array type for the additional funding sources
[ ] don't forget to update the tasklist and mutation fragments to fetch the new data
[ ] existing form_change records need to be migrated with:

[ ] a migration function that goes through every existing revision, grabs the milestone reporting requirements and creates a new form_change for each of these - and delete (archive) the form_changes for reporting_requirement, payment and milestone tables.
[ ] tests for that migration function

[ ] add the funding_agreement form to the list of database form in the server side validation plugins
[ ] fix dev data, potentially
[ ] tests for  react form and summary components need to use the same json schema as is deployed to the database.
[ ] commit_form_change_internal should use a custom handler for the funding agreement form
[ ]  tests for handle_funding_Agreement_form_change_commit
[ ] create_project_revision might need refactoring
[ ] Meets the DOD

Definition of Ready (Note: If any of these points are not applicable, mark N/A)

[ ] User story is included
[ ] User role and type are identified
[ ] Acceptance criteria are included
[ ] Wireframes are included (if required)
[ ] Design / Solution is accepted by Product Owner
[ ] Dependencies are identified (technical, business, regulatory/policy)
[ ] Story has been estimated (under 13 pts)

·Definition of Done (Note: If any of these points are not applicable, mark N/A)

[ ] Acceptance criteria are tested by the CI pipeline
[ ] UI meets accessibility requirements
[ ] Configuration changes are documented, documentation and designs are updated
[ ] Passes code peer-review
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Ticket is ready to be merged to main branch
[ ] Can be demoed in Sprint Review
[ ] Bugs or future work cards are identified and created
[ ] Reviewed and approved by Product Owner

Notes:

Input from Sepehr: this is a lot of work to do for simply moving fields around. May not be a priority at the moment.

This already happened in https://app.zenhub.com/workspaces/climate-action-secretariat-60ca4121764d710011481ca2/issues/gh/bcgov/cas-cif/1360

Fantastic! We can close it. Thanks @BCerki


Code Coverage

Codecov
Coverall

Finish with 100% codecov for hello world

Go test cannot test cannot export report in the same file


Datepicker directive: "ng-model" value must be a Date object ...

Sometimes I get the error (on input focus, TAB):
Datepicker directive: "ng-model" value must be a Date object, a number of milliseconds since 01.01.1970 or a string representing an RFC2822 or ISO 8601 date.
Have you ever met with this error?
Thank you

Can you send a plunker with example?

<input type="datetime" class="form-control" min-view="date" auto-close="true" date-time ng-model="timesheetDate" view="date">

Hi, in my html I have set the ng-model to ng-model="timesheetDate" and in my controller i have specify
$scope.timesheetDate;
however whenever I change the value of the date, it does not bind to $scope.timesheetDate
Am i missing something?

@victortey check this issue please for your case https://github.com/g00fy-/angular-datepicker/issues/90


Change warnings to python standard

Proposed changes:

Make rasa codebase conform to the python convention that warnings.warn should be used for warnings that the user needs to address while logger.warning should be used for warnings that the user should be aware of but cannot change themselves.
Changed strings that I encountered during these changesto f-strings in lieu of using "".format

See #4356
The next step will be to apply pytest.mark.filterwarnings to all those tests raising warnings.warn; but this should be merged first
Status (please check what you already did):

[x] made PR ready for code review
[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog
[x] reformat files using black (please check Readme for instructions)

@wochinge this is a WIP because it should only be merged together with additional changes, however it definitely does need review at this stage.

We can also do that incrementally. So merge this and then do another PR for the rest?

Actually this should be everything (up to the date I created this PR), I looked at every instance of logger.warning and warnings.warn in rasa for this, so everything that's not somehow marked here should be correct usage (+-)

Awesome, I think there is one more open comment, but otherwise we are good 👍

@imLew can you please update this PR with latest master and please let Tobias know when this is ready for another review


Bs dscs 539 fa copy role

10 new copy roles added finding aid 1, finding aid 2 ...finding aid 10
@scoen , @AaronCMuller , @ninhnguyen , @shuangzhou can you please look.

I have rather strong feelings about this pull request. If possible I'd like to talk about this with Kavinga and Terence.

@terenceingram , @AaronCMuller, @scoen Shaun had strong feelings too, it looks like we need to talk about it!

@shuangzhou Why did you merge this?

@bsingh-nla Can you revert this until we sort out the best approach?

@AaronCMuller I merged the change as the code looked ok, now I see that there's reason of concern for this, happy for it to be reverted.

@terenceingram we are not able to revert it from github. I'll made code change.


Add confirmation links to commands

Issue
Currently if you ban a player (/ban <name>), it executes immediately.
Solution
It would be better if the server sent a confirmation message first: Are you sure you want to ban _____?, and provide a clickable option to actually execute the ban command.

Rather than a clickable option, I've noticed a few plugins list commands that can be typed to e.g. "/yes, /no, /help"

I'm not entirely sure this is necessary, banning isn't destructive as we can always unban if it was a mistake. If we do have confirmations I'd rather it was a global /yes, /no command for everywhere that requires confirmations.


Support GPU Training

This PR supports training Cambrian-8b on GPU with deepspeed zero2. Main modification includes

We remove all the .float() that satisfies TPU's precision and use bf16 unifiedly.
We fix the llama3 chat template bug and tokenizer bug which adds additional bos token
We revert checkpoint loading and resuming impl to the default Huggingface Trainer impl
We optimize the loading of data and model

Hi @TideDra , thanks for sharing the code for GPU training. Do you also have the code for evaluating Cambrian-1? Or is it straightforward to modify the scripts from the LLaVA codebase? Thanks!

VLMEvalKit supports to evaluate cambrian.

Is there a reason why the checkpoint saving was reverted to the default HuggingFace implementation? It seems that the full model weights are stored per rank instead of the sharded model weights, so the overall size of the checkpoints is huge

Hi @TideDra , thanks for sharing the code for GPU training. Do you also have the code for evaluating Cambrian-1? Or is it straightforward to modify the scripts from the LLaVA codebase? Thanks!

VLMEvalKit supports to evaluate cambrian.

It seems to me there are some issues with the VLMEvalKit codebase when evaluating LLaVA on TextVQA. With the released LLaVA or models trained with this cambrian+gpu code, I couldn't reproduce the results reproduced in the LLaVA v1.5 paper. Not sure what's the difference between the evaluations but we probably need to modify the evaluation code from LLaVA to reproduce exact results?

@wufeim  note that we have released our eval code here https://github.com/cambrian-mllm/cambrian/tree/main/eval

@wufeim note that we have released our eval code here https://github.com/cambrian-mllm/cambrian/tree/main/eval

Oh I see it now. Thanks so much! I will check it out.
I was looking at the documentation here and thought they were not out yet. Maybe update the link in the README?

+1 the eval/requirements.txt is missing. It'd be nice to know if a specific version of datasets is needed

@wufeim @dfan sorry the requirements was masked by gitignore. added in #82

2. When I was evaluating cambrian-8b with the following command, all four GPUs are evaluating on the whole TextVQA instead of one of the four subparts. Is this correct? Or am I using a wrong command?

@wufeim have a read through run_benchmark.sh. the questions are chunked and each gpu handles one chunk.

let's move further discussion unrelated to this GPU training PR #64 to separate issues please.

Hi @TideDra,
I'm trying out the GPU training code. I see that you used zero2 for both pretraining and finetuning. Meanwhile LLaVA used zero2 for pretraining and zero3 for finetuning. I am not an expert with deepspeed but I did encounter some issues with zero3, possibly related to this. Did you have similar issues? Or how did you decide on zero 2/3?
Thanks!

Hi @TideDra,
I'm trying out the GPU training code. I see that you used zero2 for both pretraining and finetuning. Meanwhile LLaVA used zero2 for pretraining and zero3 for finetuning. I am not an expert with deepspeed but I did encounter some issues with zero3, possibly related to this. Did you have similar issues? Or how did you decide on zero 2/3?
Thanks!

In general, zero3 reduces GPU memory usage while increases training time, compared with zero2, but does not affect model perfomance theoretically. So zero2 is preferred if memory is enough. I didn't try zero3 so I didn't encounter your issue :). But actually, zero3 indeed has more bugs than zero2 in practice.

OOM when finetuning Cambrian in the 80GB A100 cluster even with a batch size of 1
Description:
I am encountering an Out-of-Memory (OOM) error while attempting to fine-tune the Cambrian model using A100 GPUs. Despite setting the batch size and gradient accumulation steps to 1, the issue persists.
Environment:
Model: Cambrian
GPU: A100
DeepSpeed Version: 0.14.4
CUDA Version: 12.1
PyTorch Version: 2.3.1
Configuration:
DeepSpeed Config: Using ZeRO stage 2 (attach zero2.json if possible)
Batch Size: 1
Gradient Accumulation Steps: 1
Mixed Precision: bf16
Steps to Reproduce:
Run the training script with the following command:
deepspeed
--num_nodes $SLURM_JOB_NUM_NODES
--num_gpus $SLURM_GPUS_PER_NODE
--master_addr localhost
--master_port 12345
--hostfile hostfile_temp
--no_ssh_check
cambrian/train/train_gpu.py
--deepspeed ./scripts/zero2.json
--model_name_or_path $ROOT_DIR/LLM/llama3-llava-next-8b
--version llama_v3
--data_path "$ROOT_DIR/cambrian/datasets--nyu-visionx--Cambrian-10M/jsons/Cambrian150K_withsystemprompt.jsonl"
--image_folder "$ROOT_DIR/cambrian/datasets--nyu-visionx--Cambrian-10M/"
--pretrain_mm_mlp_adapter "$ROOT_DIR/cambrian-models/models--nyu-visionx--cambrian-8b_projector/mm_projector.bin"
--vision_tower_aux_list '["siglip/CLIP-ViT-SO400M-14-384", "openai/clip-vit-large-patch14-336", "facebook/dinov2-giant-res378", "clip-convnext-XXL-multi-stage"]'
--vision_tower_aux_token_len_list '[576, 576, 576, 9216]'
--image_token_len 576
--num_query_group 1
--query_num_list '[576]'
--connector_depth 3
--image_position 91
--vision_hidden_size 1024
--connector_only False
--num_of_vision_sampler_layers 10
--start_of_vision_sampler_layers 0
--stride_of_vision_sampler_layers 3
--mm_projector_type sva
--unfreeze_mm_vision_tower False
--mm_vision_select_layer -2
--mm_use_im_start_end False
--mm_use_im_patch_token False
--image_aspect_ratio pad
--group_by_modality_length True
--bf16 True
--output_dir $CKPT_DIR
--num_train_epochs 1
--per_device_train_batch_size 1
--per_device_eval_batch_size 4
--gradient_accumulation_steps 1
--evaluation_strategy "no"
--save_strategy "steps"
--save_steps 500
--save_total_limit 5
--learning_rate 4e-5
--weight_decay 0.
--warmup_ratio 0.03
--lr_scheduler_type "cosine"
--logging_steps 1
--tf32 True
--model_max_length 2048
--gradient_checkpointing True
--dataloader_num_workers 4
--lazy_preprocess True
--run_name $CKPT_NAME
--report_to wandb
Behavior:
The training process fails due to OOM errors, even with minimal batch size and gradient accumulation settings.

OOM when finetuning Cambrian in the 80GB A100 cluster even with a batch size of 1
Description: I am encountering an Out-of-Memory (OOM) error while attempting to fine-tune the Cambrian model using A100 GPUs. Despite setting the batch size and gradient accumulation steps to 1, the issue persists.

How many gpu do you use? Besides we only test with vicuna-7b.

OOM when finetuning Cambrian in the 80GB A100 cluster even with a batch size of 1
Description: I am encountering an Out-of-Memory (OOM) error while attempting to fine-tune the Cambrian model using A100 GPUs. Despite setting the batch size and gradient accumulation steps to 1, the issue persists.

How many gpu do you use? Besides we only test with vicuna-7b.

2 A100 by now, for Llama3 8B.
But I only assign 1 sample per GPU. Could you give me any suggestions.

@nku-zhichengzhang it seems that 15.58G memories are reserved by pytorch but unallocated. You may follow the instruction given by the error, or try to clear cuda memories.

@nku-zhichengzhang it seems that 15.58G memories are reserved by pytorch but unallocated. You may follow the instruction given by the error, or try to clear cuda memories.

Loading the Llama3 model occupies 15GB of memory. BTW, how many GPUs did you use to train the Vicuna model?

@nku-zhichengzhang it seems that 15.58G memories are reserved by pytorch but unallocated. You may follow the instruction given by the error, or try to clear cuda memories.

Loading the Llama3 model occupies 15GB of memory. BTW, how many GPUs did you use to train the Vicuna model?

We use at least 8 gpus for pretraining and 32 gpus for finetuning. You may try zero3, which requires less memories?

@nku-zhichengzhang it seems that 15.58G memories are reserved by pytorch but unallocated. You may follow the instruction given by the error, or try to clear cuda memories.

Loading the Llama3 model occupies 15GB of memory. BTW, how many GPUs did you use to train the Vicuna model?

We use at least 8 gpus for pretraining and 32 gpus for finetuning. You may try zero3, which requires less memories?

Okay, thanks for the reply.


devops: merge from upstream and handle conflicts

COMPT-4605

conflicts were related to the README and bin/compile, bin/detect, and bin/release files

copied the latest from upstream files to our og_ files

copied from #1 for our existing implementations

Just to confirm, did you checkout changes from Heroku's repo (https://github.com/heroku/heroku-buildpack-nodejs) or from Cloud Foundry's? It should be https://github.com/heroku/heroku-buildpack-nodejs although the upstream is Cloud Foundry.

Just to confirm, did you checkout changes from Heroku's repo (https://github.com/heroku/heroku-buildpack-nodejs) or from Cloud Foundry's? It should be https://github.com/heroku/heroku-buildpack-nodejs although the upstream is Cloud Foundry.

Cloud Foundry's! I'll update accordingly.


[Skip CI] Android ndk20

Signed-off-by: Axel Nennker<EMAIL_ADDRESS>openssl supports building with the vendored feature for some time now.
sodiumoxide supports vendored builds since 0.2.2
Today zmq 0.9 support was merged with libindy and now this works.
First install an Android target:
rustup target add aarch64-linux-android
Then build libindy for Android with this command:
CC=$HOME/Android/NDK/arm64/bin/aarch64-linux-android-clang CXX=$HOME/Android/NDK/arm64/bin/aarch64-linux-android-clang++ AR=$HOME/Android/NDK/arm64/bin/aarch64-linux-android-ar cargo build --target=aarch64-linux-android
(Your mileage may vary with standalone toolchains. I used the compiler from the NDK)
No more OPENSSL_DIR, no more LIB_SODIUM_DIR etc
build.rs got simpler by just removing the checks for those environment variables.
Result:
ignisvulpis@namenlos:~/development/hyperledger/indy-sdk/libindy$ ls -l target/aarch64-linux-android/debug/
total 704128
drwxrwxr-x 36 ignisvulpis ignisvulpis      4096 Aug 26 18:42 build
drwxrwxr-x  2 ignisvulpis ignisvulpis     69632 Aug 26 18:48 deps
drwxrwxr-x  2 ignisvulpis ignisvulpis      4096 Aug 26 18:17 examples
drwxrwxr-x  3 ignisvulpis ignisvulpis      4096 Aug 26 18:47 incremental
-rw-rw-r--  2 ignisvulpis ignisvulpis 357370480 Aug 26 18:48 libindy.a
-rw-rw-r--  1 ignisvulpis ignisvulpis     12623 Aug 26 18:48 libindy.d
-rw-rw-r--  2 ignisvulpis ignisvulpis 239969038 Aug 26 18:48 libindy.rlib
-rwxrwxr-x  2 ignisvulpis ignisvulpis 123561096 Aug 26 18:48 libindy.so
drwxrwxr-x  2 ignisvulpis ignisvulpis      4096 Aug 26 18:17 native

@faisal00813 Could you please have a look? Currently I do not have a test Android app to test this with.

I wished https://github.com/hyperledger/indy-sdk/pull/1585 was merged. Then this would build

Waiting for https://github.com/hyperledger/indy-sdk/pull/1874 because feature sodium_static can be removed and should be removed. libsodium must be build with -fPIC regardless of whether it is static or dynamic. Debian packages can still be built with the static libsodium.a if needed.


Addition of information for CatFactory

Addition of information for CatFactory GitHub repository and description for project.

Thank you @hherzl!

Thank you so much for your help!


Fix JSON media type example (3.1.1)

Re-submit of PR #3434
Fix contributed by Florian Loitsch, ported to 3.1.1.md file.

also covered by #3465

I re-ran the failed check using the environment from main (see #3554) and it passed.  This is safe to merge.

Closing as a subset of #3465


Routes not loading

Based on the examples in the docs, and the tests provided with the scala play starter project, I wrote a couple tests for a new app. Unexpectedly, the requests I make in my tests come back as 404s. I'm wondering if there's some special step I need to take to ensure the routes are compiled for tests?
Here are the tests... maybe I'm missing something obvious:
import org.scalatestplus.play._
import org.scalatestplus.play.guice.GuiceOneAppPerSuite
import play.api.test._
import play.api.test.Helpers._
import play.api.libs.json.Json

class Routes extends PlaySpec with GuiceOneAppPerSuite {

  "Shotgun Routes" should {
    "find_one should be able to find onelson" in  {
      val req = FakeRequest(POST, "/sg/relay/find_one")
        .withJsonBody(Json.parse("""{"kwargs":{"entity_type":"HumanUser","filters":[["login","is","onelson"]]}}"""))
      val resp = route(app, req).get
      status(resp) mustBe OK
      contentType(resp) mustBe Some("application/json")
      (contentAsJson(resp) \ "id").as[Int] mustBe 948
    }
  }

  "DSE Routes" should {
    "sla vert labels should be able OK" in  {
      val req = FakeRequest(GET, "/dse/sla/vert-labels")
      val resp = route(app, req).get
      status(resp) mustBe OK
      contentType(resp) mustBe Some("application/json")
      contentAsString(resp) must include ("Asset")
    }
  }

}

The requests, as written, should match what I have in my routes file, and the app is working when I push requests to it in practice.
POST    /sg/relay/:methodName    controllers.RelayController.run(methodName: String)
GET     /dse/:graph/vert-labels      controllers.AtlasLegacyController.vertLabels(graph: String)
POST    /dse/:graph/gremlin          controllers.AtlasLegacyController.gremlin(graph: String)

Is there something missing?
I'm running Play 2.5.15, with scalatestplus-play 2.0.0.
Thanks,
Owen

Chatting about this on twitter, I started to wonder if there aren't extra steps required for testing routes for controllers that use DI to inject this or that. My routes depend on services provided by DI, so maybe that's where the break is. Seems really non-obvious based on the docs, however.

Hi @onelson, this should work or at least I can't see any obvious problem. Do you have the time to create a small project to reproduce this problem?
Best.

I was preparing the repro project, but in the process, I think I learned a critical piece of info.
Running tests through IntelliJ: fails.
Running tests through sbt on the command line: works.
Running tests through IntelliJ after a successful run through the command line: works.

Yeah, this is an IntelliJ problem ;)
I popped open the run configuration for the test suite and noticed some new options I hadn't seen before, such as "use sbt." When I checked the box, I started getting usable results.


changes repos links to synerbi

we probably need to manage that an old VM will have files in ~/devel/CCPPETMR_VM which will now have been changed.
This most likely also affects this wiki

What about creating a symbolic link? SyneRBI_VM -> CCPPETMR_VM

Needs repo changes in bootstrap_ubuntu.sh and possibly other files

What about creating a symbolic link? SyneRBI_VM -> CCPPETMR_VM

good idea, if CCPPETMR_VM  exists and SyneRBI_VM doesn't, create the link

update.sh needs to use ~/devel/SyneRBI_VM. Probably other things as well.

What about creating a symbolic link? SyneRBI_VM -> CCPPETMR_VM

good idea, if CCPPETMR_VM exists and SyneRBI_VM doesn't, create the link

I'm thinking that this is going to complicate the things in a consistent way for no real reason.  The important bit is to update the github links (though not fundamental either).
Wouldn't it be better to create a new VM? We may then release it with SIRF as 2.1.x.

I'm thinking that this is going to complicate the things in a consistent way for no real reason.

You mean creating the CCPPETMR_VM link? Or change update.sh? Things have to work, both for a new one and for updating an existing (but not too old) one. Not everyone is using their VMs for a short time (otherwise, why do we have the update mechanism at all?).

Wouldn't it be better to create a new VM? We may then release it with SIRF as 2.1.x.

Agreed that it would be best to create a new VM, but it seems hard to do it without a new release. how are we going to call it otherwise?

created a VM from scratch with this branch and it works. I'll try to upload it somewhere for test.

created a VM from scratch with this branch and it works. I'll try to upload it somewhere for test.

This looks ok to me now. However, I did spot a problem with the automatic set remote. On my VM, I had changed the remote manually to something else (actually git@github.com:... such that I don't need to type passwords all the time). Doing on update_VM.sh got rid of all these.
As the ccppetmr redirects are all working fine, I suggest that we remove these. People with old VMs will have no problems. People with new VMs will get the synerbi repo.
Objections? @casperdcl @paskino

I've introduced an option for the remote. Also changed the "version" stuff a bit.