Distributing tests based on number of machines

Guys, I found this article
I use Build Agent Version 2.105.2 also use  tasks DeployVisualStudioTestAgent ver. 1.0.36 and  RunDistributedTests ver. 1.0.41
I have single assembly include 100 tests  also  have group machine 4  VMs.
I set "customSlicingEnabled"

Result I see runs my test only one machine.
Please suggest what I am doing wrong.

@kardinal3000 do you see any warning in build logs?

Yep,
##[warning]Update the build agent to run tests with uniform distribution. If you are using hosted agent there are chances that it is still not updated, so retry using your own agent.

We have identified this as a bug in core clr (I assume you are using coreclr agent). if you are using any runsettings/testsettings we might be able to give a workaround till we get a fix. Let me know if you are using runsettings/testsettings

Hi acesiddhu.
I want to describe more deeply use case with distributed test execution.
We are using ms test framework.
We have 100 Web driver tests in one assembly.
We want run tests in group of test agents, using test plans.
Will this approach work for ui webdriver tests ?
The main idea to decrease time of webdriver test execution.
Also as far as I understand we can decrease time of running non-ui tests(unit tests, rest api functional tests, integration tests) by adding parallelization params in testsettings file.
But this approach with testsettings file will not work for UI tests.
Am I right or I missed something ?
Maybe you can suggest us something?

Thanks for answer! I hope you will fix it, as soon as possible. Yes we are using runsettings/testsettings on the some parts of tests. I will be appreciated if you shared your workaround.
But, usually we use Test Agent and RunDistributionTests tasks together. Probably your workaround not be useful, just my thoughts

The issue will be fixed in next coreclr agent release. Till then if you want to workaround this you need to add this in your settings file
Runsettings: add the bold part in your runsettings

****

TestSettings: add the bold part in your TestSettings
<TestSettings name="TestSettings1" id="5e7c73a5-8e38-4d1e-a402-459dfab945b6" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010\">
<Execution hostProcessPlatform="MSIL">
<CustomSlicing enabled="true" />

I suggest if you want to try the workaround make a temporary copy of the setting files and use that in Build flow till new agent is released.

@acesiddhu  just clarify  in which part I should add the bolt part? and what does it mean bold part
"
"
I see empty part.
This is my TestSettings1.testsettings
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings
  id="5757f364-edaf-4350-9847-1da89fc47b76"
  name="TestSettings1"
  enableDefaultDataCollectors="false"
  xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description><!--_locID_text="Description1"-->These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
</TestSettings>

Can you try this
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings
  id="5757f364-edaf-4350-9847-1da89fc47b76"
  name="TestSettings1"
  enableDefaultDataCollectors="false"
  xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description><!--_locID_text="Description1"-->These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
<Execution>
<CustomSlicing enabled="true" />
</Execution></TestSettings>

Thanks a lot for workaround!
But I observe strange behavior. I use TFS on-premise. I have two VMs (with two test agents TFSTA8 and TFSTA7) when I run my tests they are running only one VM but twice. The second VM did not used

What could it be?
Use BA 2.105.5

After adding CustomSlicing to RunSettings we are getting the error below.  If I don't specify the runsettings file then tests run fine but don't distribute across machines.  We also get this error when we try to run using TestPlan instead of TestAssembly.  Any guidance on what to check?
2016-09-08T16:04:02.9480498Z ##[warning]Update the build agent to run tests with uniform distribution. If you are using hosted agent there are chances that it is still not updated, so retry using your own agent.
5 2016-09-08T16:04:03.0886726Z DistributedTests: Environment WinRm Protocol HTTP.
6 2016-09-08T16:04:03.0886726Z DistributedTests: Trying to unregister TestAgents from Test Machines
7 2016-09-08T16:04:03.1667965Z DistributedTests: Task 'UnRegisterAgent' on machine 'spo-iap-auto-03.itron.com:5985' is being run
8 2016-09-08T16:04:03.1667965Z DistributedTests: Task 'UnRegisterAgent' on machine 'spo-iap-auto-02.itron.com:5985' is being run
10 2016-09-08T16:04:03.1667965Z Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.ArgumentNullException: Value cannot be null.
11 2016-09-08T16:04:03.1667965Z Parameter name: deploymentMachine.Credentials.UserName
12 2016-09-08T16:04:03.1667965Z    at Microsoft.VisualStudio.Services.DevTestLabs.Definition.ValidationHelpers.ValidateNullOrWhitespaceParameter(String paramName, String paramValue)
13 2016-09-08T16:04:03.1667965Z    at Microsoft.VisualStudio.Services.DevTestLabs.Deployment.Deployment.DeploymentClient.d__15.MoveNext()
14 2016-09-08T16:04:03.1667965Z --- End of stack trace from previous location where exception was thrown ---
15 1016-09-08T16:04:03.1667965Z    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
16 2016-09-08T16:04:03.1667965Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
17 2016-09-08T16:04:03.1667965Z    at Microsoft.VisualStudio.Services.DevTestLabs.Deployment.Deployment.DeploymentClient.d__14.MoveNext()
18 2016-09-08T16:04:03.1667965Z    --- End of inner exception stack trace ---
19 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
20 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
21 2016-09-08T16:04:03.1667965Z    at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.DtlMachineDeploymentHelper.<>c__DisplayClass33_0.b__0(KeyValuePair2 deploymentTask) 22 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Parallel.<>c__DisplayClass42_02.b__1()
23 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
24 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object )
25 2016-09-08T16:04:03.1667965Z    --- End of inner exception stack trace ---
26 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
27 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
28 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner1 source, ParallelOptions parallelOptions, Action1 simpleBody, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally)
29 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally)
30 2016-09-08T16:04:03.1667965Z    at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source, Action1 body)
31 2016-09-08T16:04:03.1667965Z    at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.DtlMachineDeploymentHelper.WaitTillCompletion(Dictionary`2 taskList, String friendlyNameOfTask, Boolean addToErrorMachines)
32 2016-09-08T16:04:03.1667965Z    at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.DtlMachineDeploymentHelper.<>c__DisplayClass31_0.b__0()
33 2016-09-08T16:04:03.1667965Z    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
34 2016-09-08T16:04:03.1667965Z    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
35 2016-09-08T16:04:03.1667965Z    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
36 2016-09-08T16:04:03.1667965Z    at System.Threading.ThreadHelper.ThreadStart()

@kardinal3000 the new agent is released https://github.com/Microsoft/vsts-agent/releases/tag/v2.105.6 can you try with that. you can undo the changes you did in runsettings and testsettings

@geoffstew4 The error is not related to this. Can you try using the new agent that is released https://github.com/Microsoft/vsts-agent/releases/tag/v2.105.6 and undo the changes you might have done in runsetting/testsetting

@kardinal3000 Did your issue get resolved after using the build agent shared by @acesiddhu .

@SatishThareja I used the last version Build Agent 2.106. And Unfortunately, I have the same issue as  I wrote above.
I observe strange behavior. I use TFS on-premise. I have two VMs (with two test agents TFSTA8 and TFSTA7) when I run my tests they are running only one VM but twice. The second VM did not used

@kardinal3000 can you share the following logs and send it to<EMAIL_ADDRESS>
build logs
c:\Dtalogs\DtaExecutionHostlog from both agent machines (clearing off the logs for previous runs)
Addtionally can you let us know if:
Same tests are executed twice or is it a different set?
Are both machines used if you do not enable Custom Slicing?

@kardinal3000 Please share the logs to help us debug this issue.

Closing this issue due to no response.


feat(frontend): support show internal tables

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
This section will be used as the commit message. Please do not leave this empty!

Support show internal tables.

Example:
dev=> show internal tables;
                   Name
------------------------------------------
 __internal_v_20_hashjoinright_1019
 __internal_v_20_hashjoindegreeleft_1018
 __internal_v_18_grouptopnnode_1011
 __internal_v_20_hashjoindegreeright_1020
 __internal_v_17_topnnode_1010
 __internal_v_19_hashjoindegreeleft_1014
 __internal_v_20_hashjoinleft_1017
 __internal_v_18_hashaggresult_1012
 __internal_v_19_hashjoinright_1015
 __internal_v_19_hashjoindegreeright_1016
 __internal_v_19_hashjoinleft_1013
(11 rows)

dev=> select * from __internal_v_19_hashjoinleft_1013 limit 5;
 orders.o_orderkey | orders.o_orderdate | orders.o_shippriority | customer.c_custkey | orders.o_custkey
-------------------+--------------------+-----------------------+--------------------+------------------
                69 | 1994-06-04         |                     0 |                 85 |               85
               256 | 1993-10-19         |                     0 |                125 |              125
              1154 | 1992-02-15         |                     0 |                 37 |               37
              1792 | 1993-11-09         |                     0 |                 49 |               49
              1894 | 1992-03-30         |                     0 |                 76 |               76
(5 rows)

Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.

SQL commands, functions, and operators

Release note

Support show internal tables statement. Developers will be able to explore the state implementation of RisingWave in depth with this statement.

Refer to a related PR or issue link (optional)
https://github.com/risingwavelabs/risingwave/issues/7304

Just curious, querying from this will query the StateTable via a MaterializeExecutor is this correct?

Just curious, querying from this will query the StateTable corresponding to the internal table via a MaterializeExecutor is this correct?

No, it will query from storage layer directly. To be more precise, read from the shared buffer of the compute node(s)


Default input value disappears on setValue(getValue())

This might be just an edge-case, not really an issue with the react-form itself. I still prefer to report this as I am not sure how can I solve it.
It seems like values provided by <Form values={} /> are not available through getValue() in FormInput on the "first run".
The input value is set to undefined when I am trying to setValue( getValue() ) in the onBlur callback without changing the input default value (it works when i change the value). I believe it should set the input value to its default value provided by <From values={ ... } /> instead of undefined.

Story:
I've created a custom input in which I verify the given value (in onBlur) and eventually change it. For example if the input value is 12 it will change to 12:00, 1:3 becomes01:30 and so on:
Custom input: <HourInput />
import moment           from "moment";
import React            from "react";
import { FormInput }    from "react-form";

export default function ( { field, isForm, ...rest } ) {
    return (
        <FormInput field={field} isForm={isForm}>
            { ( { setValue, getValue, setTouched } ) => (
                <input {...rest}
                    value={getValue()}
                    placeholder="00:00"
                    onChange={ e => setValue( e.target.value ) }
                    onBlur={ () => { setValue( verifyValue( getValue() ) ) } }
                />
            ) }
        </FormInput>
    )
}

function verifyValue( value ) {
    const time = moment( value, "HH:mm" );
    return time.isValid() ? time.format( "HH:mm" ) : "00:00";
}

I am rendering those inputs in a loop, like so:
class Foo extends Component {
    [...]

    renderHoursInputs = period => {
        const verb  = period === "start" ? "Start:" : "End:";
        const hours = MenuUtils.parseHours( RestaurantStore.data.hours );

        return (

                {verb}

                { Object.keys( hours ).map( day => (
                    <td key={`${period}_${day}`}>
                        <HourInput field={[ "hours", day, period ]} />

                ) ) }

        );
    }

    render() {
        const hours = MenuUtils.parseHours( RestaurantStore.data.hours );
        // hours = { day_index : { start : hour, end : hour }, ... ]
        // hours = { 1 : { start : "08:00", end : "16:00" }, ... }

        return (
            <Form onSubmit={...} validate={...} values={ { hours } }>
                [...]
                {this.renderHoursInputs( "start" )}
                {this.renderHoursInputs( "end" )}
                [...]
            </Form>
    }
}

The default value specified in <Form values={...} /> matches the array-like field name and is displayed correctly as you can see:

When I click on the input and change it's value, everything works correctly. The input value is changed and eventually auto-corrected as intended.
The problem appears when I click on the input and then blur it without changing anything. Then it's value is changed to 00:00 (the default one returned by verifyValue), even if the input had a default value set by the <Form />:

I've tried to use some kinds of hacks, for example hard-setting the value directly in the input, but then the input becomes controlled (?) and I cannot change it's value anymore:
// rest.value = default value
<input {...rest} value={getValue( rest.value )} />

Any thoughts on how can I solve this issue? I am not sure if I am doing something wrong or if there's a problem with the way react-form handles defaults values provided by <Form />.

Using the latest Node.js and react-form version.

I am trying to debug this with React Devtools and I've found something strange.
Even if every input has a default value (which are by the way displayed correctly, see picture), only the first one has a value attribute in the component. When I click on another HourInput, a value property appears for the input I clicked on but dissapears on all others:

The first one has it's value set correctly:

This one should have value="23:59", but has nothing set:


Confusion about AsyncActionCreator and ActionCreator's accesibility

First: huge thanks for this, it's super interesting!
I came across the problem that in the current released version (the one that cocoapods picks up), AsyncActionCreator and ActionCreator are not public typealiases, which was super confusing at first. Would it worth to release a new version soon? I saw this has been fixed since then.
Cheers!

okay, maybe I'll take it back, it seems like I still can't reference these outside of the framework.
I'll post details soon.

Okay, so I figured out, I have to reference the typealias like this:
Store.AsyncActionCreator
The reason why I got confused is because this wasn't mentioned anywhere:). to be honest, I expected a typealias to be public not on a protocol but as "free typepalias".
Is there any specific reason why you wouldn't want to use the ones on StoreType simply as "standalones"?
I mean these:
typealias ActionCreator = (state: State, store: StoreType) -> Action?

/// AsyncActionCreators allow the developer to wait for the completion of an async action.
typealias AsyncActionCreator = (state: State, store: StoreType,
actionCreatorCallback: ActionCreator -> Void) -> Void

My confusion was elevated by the fact when I redefined both of these in my project, I got a compiler segfault.

I guess it's useful to specific the Store generic in the signature to avoid casting and that's why its implemented this way. I'm happy to translate some of the async flow documentation to ReSwift from the original redux docs to make this clear. Where would be a good place to contribute it?

Hey @itchingpixels! You figured everything out before I could respond - awesome :smiley:
An update to the docs would be very much appreciated! :+1:  I think it would be best to add a new markdown document to this folder: /ReSwift/Readme/Async Actions (or something similar).
Once the PR is merged we will regenerated the docs hosted at http://reswift.github.io/ReSwift to include that chapter.
You also brought up a good point of making a new release. There have been a lot of changes since the last one. I'll take care of that!
Thanks a lot for the contribution :tada:

Either I haven't read the docs thoroughly, or it's still not clear how to properly make an ActionCreator.
This is not clear at all
func deleteNote(noteID: Int) -> ActionCreator {
   return { state, store in
       // only delete note if editing is enabled
       if (state.editingEnabled == true) {
           return NoteDataAction.DeleteNote(noteID)
       } else {
           return nil
       }
   }
}

What's NoteDataAction.DeleteNote(noteID)?
Something like this also doesn't work:
let loadMainSequence: Store.ActionCreator = { (_ state: AppState, _ store: Store<AppState>) -> Action? in
    return %someAction%
}

I can't store.dispatch(...) it, because Cannot convert value of type 'Store<AppState>' to expected argument type 'Store<State>' (WAT)

Well, I figured it out, I think. For example:
func loadMainSequence() -> Store<AppState>.ActionCreator {
    return { state, store in
        ...
        return SetRouteAction([AppRoute.login.string])
    }
}

store.dispatch(AppRouter().loadMainSequence())

This is kinda not obvious from the docs.


[Feature]: add specifications for interop demo

Before request a new feature

[X] I have searched the issues of this repository and believe that this is not a duplicate.

Describe the feature you'd like supported
the contents under original demo directory is always misunderstood, redefine it as interop
Describe alternatives you've considered
No response
Additional context
No response

Finished task.


Monaco works alone under node env with vscode-textmate.

This pr is for using Monaco-editor with vscode-textmate. I use the setTokensProvider api for this, but while now the setTokensProvider can only accept string tokens, and actually in monaco there is a api called _toBinaryTokens to transform the string tokens into Uint32Array, while this api can only accept scopes as a string not a string array which is given from JsGrammar.tokenizeLine. So I changed the api for setTokensProvider and make it accept a tokenizeLine2 function to use the Uint32Array tokens directly without _toBinaryTokens. The way to do what I want is below:
// get themes from vscode extensions
import { Registry, StackElement } from 'vscode-textmate';
const registry = new Registry();
registry.setTheme({ name: 'Dark+ (default dark)', settings: (DarkTheme.tokenColors.concat(DarkThemePlus.tokenColors)) });
const JsGrammar = registry.loadGrammarFromPathSync('./JavaScript.tmLanguage.json'));
const jsConfig = require('./javascript-language-configuration.json');
languages.register({
  id: 'javascript',
  extensions: ['.js', '.jsx'],
});
languages.setLanguageConfiguration('javascript', jsConfig);
languages.setTokensProvider('javascript', {
  getInitialState: () => {
    return StackElement.NULL;
  },
  tokenize2: (line, state) => {
    const rule = JsGrammar.tokenizeLine2(line, state);
    const ruleStack = rule.ruleStack;
    return {
      tokens: rule.tokens,
      endState: ruleStack,
    };
  },
});

@wenshiqi0,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla.microsoft.com.
It will cover your contributions to all Microsoft-managed open source projects.
Thanks,
Microsoft Pull Request Bot

@wenshiqi0, thanks for signing the contribution license agreement. We will now validate the agreement and then the pull request.
Thanks, Microsoft Pull Request Bot

This can be closed.

Any reason this feature wasn't left in?


Ensure swipeTolerance is taken into consideration for the Y axis

When swiping on the Y axis the swipeTolerance option is disregarded causing events to fire no matter the distance swiped. X axis works as expected.

I have tried this for so many times, but it works well

I am also experiencing this bug: tolerance only affects X, not UY swipes. This is on iPad air 2, iPadOS 13.

I have forgotten this issue for a long time. Sorry about that.
Thanks for @cameronbatty 's contribution. I have merged this PR and release a new version of v3.0.1. Thank you


Installation and uninstallation of extensions

During the use of the browser, the user can uninstall any extension. After uninstalling, the user can actively go to the Chrome Web Store to reinstall the extension, or install the browser again to restore the extension.

19822709e00d20e9392a9dcd0c7f47f7ed4e20f1


CI housekeeping

Splits our single workflow into two. This means that we can use on: pull_request, which works much better™  with other actions (like checkout).
It also means (I think) that PRs which themselves modify the workflow(s), should be exercised in the PR (unconfirmed, but I think it should work like that), which grant has been running into a few times recently...
The rationale for on: pull_request_target was that the github token given to on: pull_request does not have enough permissions to read repo secrets and things, so it could not deploy. This is not so with pull_request_target.
By splitting deploy into on: workflow, the token its natively given has the appropriate permissions.
The image from the build job is shared with the test job in the same workflow as an artifact. It is not shared directly (e.g. as an artifact) with the deploy workflow, but the build steps use a shared cache (see cache_from and cache_to), which achieves basically the same result.

I have two additional commits here which we can also look to add in the future if wanted

I tried this out on my branch. It seemed to work, including against my new get_service_ip test.

Going to merge this now then, I think it's working too, and should be quite a big improvement to be actually testing the PR branch code!

looks like the CI on this PR ran twice (pull_request and pull_request_target?) but dont see that on the merge to main


Depth loss not mentioned in the paper and seems heuristic

In your paper Sect 3.2, it reads like you only applied three losses - mask, reg and mask. However when looking at the code, you seem to use GT depth to additionally supervise the training. The depth supervision is further set to True to brandenburg_gate.yaml and palacio_de_bellas_artes.yaml, but not lincoln_memorial.yaml and pantheon_exterior.yaml.
In your reported results, do you use GT depth supervision? If so, why is it True for two cases and False for others? Are there any reasons behind this kind of heuristics?

Hi, thanks for your interest in our work!
We did use depth supervision, but not the  ground truth depth from LiDAR scans. The depth used for supervision is from SfM 3D points reconstructed by LoFTR. We load the depth in here, and use it for depth loss durning training. The name depth_gt is indeed a little confusing, but it's just to distinguish supervision depth from prediction depth.
We use depth supervision on scene Brandenburg Gate and  Palacio de Bellas Artes, due to the fact that imperfect semantic maps will degrade the reconstruction quality of thin structures. We empirically found that  sparse SfM  depth from LoFTR can fix this problem.
We will add details about depth in the  supplement material, thanks for pointing it out.


add missing sip_remove_header()

This is the counterpart to sip_add_header()

@guss77 Looks good! Would you mind also adding a test similar to the one for sip_add_header?

This is very nice, thank you!

:+1:

Hey guys, I understand there's no activity on this repo? I'm cleaning up my stuff and would like to remove my fork with the PR, as I'm not using Adhearsion anymore.
If you still want to merge this PR, please do, otherwise I see no reason for me to maintain it.


Bump e2e to use latest CAPI/CAPV

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

/test pull-cloud-provider-vsphere-e2e-test

/test pull-cloud-provider-vsphere-e2e-test
/test pull-cloud-provider-vsphere-e2e-test-on-latest-k8s-version


Could not connect to Server iOS 9.+

Hi,
I got an issue sending a POST request to our API when using Alamofire on iOS 9.1 +, minor iOS versions seem to be fine and the problem occurs afaik on several versions of Alamofire (2.0, 3.0 - 3.4 tested).
I created a simple sample code to test this case:
In detail my first request I send always fails with "Could not connect to server":
Alamofire.request(.POST, "https://...", parameters: parameters, encoding: .JSON, headers: nil)             .authenticate(user: "<user>", password: "<password>")             .responseString { (response: Response<String, NSError>) in                 print(response)         }

FAILURE: Error Domain=NSURLErrorDomain Code=-1004 "Could not connect to the server." UserInfo={NSUnderlyingError=0x7f7f90e12cc0 {Error Domain=kCFErrorDomainCFNetwork Code=-1004 "(null)" UserInfo={NSErrorPeerAddressKey=<CFData 0x7f7f90e35f60 [0x1062637b0]>{length = 16, capacity = 16, bytes = 0x100201bb59bf42510000000000000000}, _kCFStreamErrorCodeKey=-2200, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=, NSErrorFailingURLKey=,n, _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-2200, NSLocalizedDescription=Could not connect to the server.}

The FAILURE is thrown immediately and there's no incoming request on our server side, at least the error message is true in this case.
Sending the same request again once the application was started works like charm, furthermore using the NSURLConnection approach works fine aswell, even on startup:
`        let req = NSMutableURLRequest(URL: NSURL(string: "https://...")!)
    req.HTTPMethod = "POST"
    req.setValue("Basic ...", forHTTPHeaderField: "Authorization")
    req.setValue("application/json", forHTTPHeaderField: "Content-Type")

    req.HTTPBody = self.reqString.dataUsingEncoding(NSUTF8StringEncoding)

    let data = try! NSURLConnection.sendSynchronousRequest(req, returningResponse: nil)

    print(String(data: data, encoding: NSUTF8StringEncoding))

`
The only clue I have so far is that this is the only POST request in my app that uses Authorization atm, the others do their job as expected.
I'm grateful for any hint pointing in the right direction or if anyone faced similar problems and I'll will continue debugging for a while till I decide to implement a workarround.
Thx

Update: It's not the Authentication, removed it from the logic and still getting the same behaviour

Hi @marxcheckito,
These types of questions are best suited for Stack Overflow. We use our GitHub project or bug reports and feature requests.
Best of luck! 🍻

From our Contribution Guidelines
Asking Questions
We don't use GitHub as a support forum. For any usage questions that are not specific to the project itself, please ask on Stack Overflow instead. By doing so, you'll be more likely to quickly solve your problem, and you'll allow anyone else with the same question to find the answer. This also allows maintainers to focus on improving the project for others.

I am getting the exact same error. It happens on the first request only. After that it works like a charm. Its weird. Tested on device and simulator. It always fail the first request. Why closing this @cnoon? Maybe a hint to swift newbies like me?

You can try the sample app I just created for a rdar @cescoferraro here. That makes a request with Alamofire upon launch. You can see if you see the issue there...

@cnoon oh! Got it!
So for now pod 'Alamofire', 'https://github.com/Alamofire/Alamofire.git'  is a workaround?

#1 The request I'm doing is not triggered upon launch (App is running for quite a while and I even do some GET requests before that)
#2 I only get the bug on POST requests so as far as I see I don't even need to try this sample.
#3 As I don't have the problem with iOS 8.1, or NSURLConnection or NSURLSession I consider it a bug and thought you might wanna know about it.
FYI I implemented a retry counter which ignores the first (n) -1004 errors (according to my settings) and actually it works fine after the first request was discarded.

Last comment:
Was able to reproduce the behaviour using NSURLSession approach (iOS 8 its fine iOS 9.0 first request fails) so it's not an Alamofire issue.
Thanks :)

Thanks for reporting back @marxcheckito. 🍻

I am getting the exact same error. I installed pod as was supposed  pod 'Alamofire', 'https://github.com/Alamofire/Alamofire.git'
though that did not change the thing.
I get the bug on GETrequests every time I push start button.  It happens on the every request. So i completely stuck with it, donnu know what to do.
Could you help me?

Same here. None of the solutions helped.

I found solution.
In Terminal try  ifconfig.
i f you have ip address <IP_ADDRESS> - change you local agress to this one.

Hey guys!
Seems like the core of this issue is in nginx, as described here:
http://stackoverflow.com/questions/36907767/nsurlerrordomain-code-1004-for-few-seconds-after-app-start-up
There is a release of nginx, that fixes underlying issue - 1.11.0, however it's not stable, so we might have to wait until it's released.
For now switching to HTTP 1.1 seems like the most sane solution to this problem.

Great we suspected some problems with our nginx but couldn't find the issue yet. Might have a look into that.

We also have the same issue, and are also using nginx. Though the really strange part of this, is that it only occurs on iPhones. We've tested it on iPod's running iOS 9.3 and iPads running iOS 9.3 and on those devices everything is working perfectly fine.

setting the encoding parameter worked for me  Alamofire.request(.GET, requestUrlString, encoding: .JSON).responseJSON { response in } .If i don't set encoding parameter. i am getting above error.


Add CI for both parts, smart contract and UI

Context
This is a mono repo that will host all the code.
Goals
To setup CI for:

The frontend code
The Rust  smart contracts we are going to develop onwards.

Alright ! Here is the initial CI for the rust contracts ! https://github.com/eigerco/nebula/pull/22
Moving this for review :fast_forward:


cli to pcap

CLI reads payloads like LogStream does today, and writes PCAP.

Spike if we get resources.


502 Bad Gateway on all requests

Server is responding with 502 Bad Gateway on all requests.
Started around 4pm EST (1/8/2020).

Just ran into this as well,
The only request I'm not getting 502 Bad Gateway seems to be for top anime by popularity
https://api.jikan.moe/v3/top/anime/1/bypopularity

The issue has been resolved, thanks for reporting. Please re-open this issue if you face the problem again.


Sitemap generation

Generates the sitemap and puts it in a local file.  Next PR will push the file to S3.  Then there will be packaging for execution in Lambda.

Just two questions.


Not all memories showing up on Memory Link List

I have been able to access only six out of the eight total Memories associated with Memory Link. I am unsure of why I am unable to access the following Memories: "A Triple Team" and "Taking It to the Next Level." Here are the issues I noted during my Memory Link experience:

I was able to get the Memory "Different Bones and Dreams" to activate and play out, but I did not have it listed on the Memory Link List until after completing the Memory.
I was unable to get the Memory "A Triple Team" to activate and play out. I participated in the double battle with the triplets in case that possibly changed the dialogue and / or activated the Memory to try and troubleshoot the issue.
I was unable to get the Memory "Taking It to the Next Level" to activate and play out. I could not find Skyla anywhere in front of the Mistralton City Gym as noted in the Memory description on the Memory Link list.

I was able to get all remaining Memories to activate and play out (i.e., "Hearts' Desire," "A New Light," "White and Black," "Farewell, Beloved Sister," and "Meeting Friends, Saying Good-Bye"), and I am in the process of capturing N's Pokemon. The Game Sync ID is from a game that has a completed save file for White. I also tried reconnecting to Memory Link via Nintendo WFC after verifying the same Game Sync ID and the DNS noted by Entralinked. It does not update the Memory Link List nor does it activate any Memories any-game.
If you are able to shed some light on this issue I am experiencing, I would greatly appreciate it. Thank you!

debug.log

I am unable to reproduce this issue. Are you sure that you meet the criteria for the events?
Have any of the save files been tampered with perhaps?

To activate "A Triple Team" you have to talk to each of the Gym Leaders and then talk to Cilan.

Yes, I believe I meet the criteria for both Memories, but I cannot get those two Memories to activate and play out. The original Game Sync ID that I used that produced those issues above was confirmed to have a save file in which the player completed the main story in Pokemon White. Similarly, that player provided me with a second different Game Sync ID from a save file in which the player completed the main story in Pokemon Black. This Pokemon Black save file (i.e., the second Game Sync ID) has a Gold Trainer Card if that helps. That same player who provided me with the first two Game Sync IDs also provided me with a third Game Sync ID from a save file in which the player completed the main story in Pokemon Black. I can also confirm that all three Game Sync IDs are valid and allow me to obtain Memories, but the Memory Link List never updates. I am able to confirm this since I was able to activate later Memories (according to the progress in the save files for Pokemon White / Pokemon Black), such as "Meeting Friends, Saying Good-Bye," which requires you to beat N and Ghetsis at the end of the main story.
No, I do not believe the save files from any of the games have been tampered with (i.e., Pokemon White with the first Game Sync ID, Pokemon Black with the second Game Sync ID, Pokemon Black with the third Game Sync ID, Pokemon White 2 [my personal copy]). The individual providing me with all three Game Sync IDs from their copies of Pokemon White and Pokemon Black has not tried accessing Memory Link via Entralinked, and others I have discussed this issue with have also confirmed not accessing Memory Link via Entralinked. These players have mentioned only doing so locally.
I am not 100% sure, but I believe the issue stems from either my personal copy of White 2, which I want to mention I have not had any issues anywhere else (including normal gameplay), or the Entralinked program itself. I have no way of testing to see if my issue would be resolved if I connected locally to Pokemon White / Pokemon Black.
debug.log

Maybe I'm missing something obvious but I can't think of a possible cause.
Would you be willing to upload the save files so I can try and debug the issue?

I don't mind. Can you please provide me with instructions on how to upload the save files?


Improve dark mode colors

Some contrasts are really inconsistent and hard to differentiate

Some screenshots of bad examples would be great

Modal blur has not enough contrast for dark mode + in these examples the gray button too light & the background of the modal dialog is also too light:

Gray buttons here are better but don't know if they should be even darker, same for orange:

Not a UX Expert tough, just my feelings.

In addiiton, the "black" buttons in Home (Send/Receive) are too dark in dark mode => need to be lighter.


[Bug]: Extra "" needed around password input value for API body JSON

Is there an existing issue for this?

[X] I have searched the existing issues

Description
I have an API that accepts the body in this format:
{
    "email":"{{Input_email.text}}",
    "password":"{{Input_password.text}}"
}

However, when I try to call it with variables, it throws errors (saying the password needs to be a string). I checked the body sent with the AppSmith request, it showed that it was actually sent like this:
{
<EMAIL_ADDRESS>    "password":123
}

As you can see, the password "" was stripped. I was only able to resolve this after modifying the API JSON body within AppSmith to include an extra "" around the password input value like this:
{
    "email":"{{Input_email.text}}",
    "password":""{{Input_password.text}}""
}

I thought that such behavior is odd and could be a bug!
Steps To Reproduce

Add a Rest API that requires Raw JSON in the body
Replace variable values with input fields with the password field being masked

Public Sample App
No response
Version
Self-Hosted Appsmith v1.6.18

@aaahariri you can disable smart substitution for the API in the settings to solve this

@Nikhil-Nandagopal
Thanks for your response. I actually want to use smart substitution but the behavior is not consistent at the moment. It removed the "" i only one of the smart tags.

Further to the above, it seems that if the input was just numbers, the "" would be stripped but not if the input includes letters and is treated as a string.
The same API/Query would throw an error if the input didn't include any letters (and therefore was treated as a number). However, if the input to the same field was a string (numbers and letters) it would not throw any errors and would work fine.
I don't think such inconsistent behavior is normal.

@aaahariri smart substitution is used to automatically add the quotes around strings and automatically escape special characters such as apostrophe's inside them. This means that the code looks at the key in the JSON and detects if it is a number and removes the quotes and treats it as a number while if it is not a number, it adds the quotes and escapes any special characters


Naive search and replace of => to -> change

Attempt at implementing #36
EUnit tests pass FWIW!

Great, thank you for this.
Could you also update the syntax in the .gleam files? A search and replace should work fine as the => token has been removed entirely


Error loading more table items when using viewer metadata

Hi,
First of all, thanks for Clerk, its awesome ;)
I've encountered bug, that loading more items in table does not work, when using viewer metadata:
;; ## Given
(def numbers (into [] (for [a (range 100)]
                        [a])))

;; ## Click 'more' for this table works...
(clerk/table numbers)

;; ## .. but clicking more here causes "Error: No protocol method IStack.-peek defined for type object"
^{::clerk/viewer clerk/table}
numbers

I'm using io.github.nextjournal/clerk {:mvn/version "0.5.346"}

This has been fixed in v0.12.707:
https://github.com/nextjournal/clerk/blob/0abef98cecc7e8d5528571902f67e7e54bb67f54/CHANGELOG.md?plain=1#L379


Automate artifact generations

Add a workflow to generate and upload JAX model artifacts.
Currently there are many models can't be generated due to errors (#125), so this workflow is just for demonstration and can be enabled once we fix the workload.
It also enhances the JAX artifact generation script. Similar changes will be applied to other frameworks in the follow-up changes.
Tested on https://github.com/openxla/openxla-benchmark/actions/runs/6216252740/job/16869918561. The upload step is failed as expect because presubmit runners don't have permissions

Kindly ping : )


pip install doesn't seem to work without providing a version number

"pip install django-production-ready" doesn't seem to work.

It works now.
Earlier there was only alpha releases on pypi, probably pypi doesn't install alpha releases unless they are pinned.


Updates in facerec_from_webcam_faster.py

Feature:

Updated code to fetch multiple images for recognition from directory (any number of images).
Added 'images' directory.

Note:

Updated code with fetching from directory feature.
Multiple images can be placed in the 'images' directory and can be recognized automatically.
Recognized face will have the name of the image file (without extension).
Supported image extensions JPG, PNG, JPEG.

Hi @ageitgey  can you please review and add these little changes.
I am new to Python and OpenCV, it will be a motivation for me if I can contribute to your project.

Hi, all your updates seem reasonable. But I am trying to keep the sample programs hereas simple as possible because they are just meant to be a demo of how to call the functions of this library and not meant to be really useful programs on their own.
Your script is a more useful program for sure, but it might be better as it's own github repo or something like that. But great job on the code!


Add boolean input variables to reevaluate primary cache key & to only restore (skip save post run)

I got a little ambitious with this one. Understand it may never get merged. This tackles a few PRs / issues & topics of conversation all in one:
On the topic of skipping the post run save script, aka only restoring:

#489

Instead of an environment variable, I add the input variable only-restore. Default is set to false.

On the topic of reevaluating the primary cache key:

#673
#784

For this, I add the input variable reeval. Default is set to false.

Being that I added new input variables, I had to do quite a bit to get all checks to pass. The goal was adding reevaluation of the primary key during the post run save script as a capability, which was accomplished. In a nut shell, for reevaluation, you need logic to handle when the primary key gets saved to the cache's state. This PR handles doing that.
The jest tests were edited bare minimally to pass. New jest tests were not added for say, testing setting boolean inputs of only-restore and reeval to either true or false. However, test utilities were added to be able to read these input boolean variables. Also, Inputs.Reeval was set to false throughout both save and restore jest tests to evaluate save & restore scripts appropriately with the default reeval action input value.
The 'Tests' workflow was however changed to begin to handle setting the boolean variables to either.
The new field of reeval was added to the job matrix of the test-save & test-restore jobs. In addition, matrices were added to the test-proxy-save & test-proxy-restore jobs to handle this input as well. To differentiate primary keys between matrix runs, a -${{ matrix.reeval }} partial was appended to these keys.
In addition, to handle the only-restore case, new jobs of test-only-restore & test-proxy-only-restore were created, each setting reeval to true, and their setting of the primary key follows the convention described above.
I created a tag on the main branch of my own repo with these features as well. I've also tested it in quite a few other repos where it's used within composite actions.

<EMAIL_ADDRESS>
Examples using the prncevince/r-actions/setup-knitr-cache composite action:

reeval set to true:

prncevince/r-actions/.github/workflows/knitr-cache.yaml
prncevince/test-workflow-knitr-cache/.github/workflows/knitr-cache.yaml

only-restore set to true:

prncevince/r-actions/.github/workflows/restore-cache.yaml

I also added to the README.md & to examples.md. To the readme I added descriptions of the input variables with their defaults. To the examples I added an example for using knitr caching when rendering R Markdown, as well as detailed explanation of why you would want to use it, with links to examples mentioned above.
A lot of this implementation could be better. I think that the creation of the primary keys during the Tests workflow could be based on an actual file hashing. This could be checked into the repository - say the __test__/__fixtures__/helloWorld.txt). Then, edits to the file could be reflected in the primary key and indeed be tested upon reevaluation.
I'm sure better messaging could be implemented with only restoring (forgoing the post run save script). Not sure that I'm going to pursue this much. Full implementation works right now on my own tag. But if what's requested by reviewers is simple enough for me to have a go to get it merged, I may be interested.
Cheers

Hi @prncevince 👋🏽
Thank you for the well-thought contribution..
We have had a few similar feature requests/ PRs in the past.
https://github.com/actions/cache/pull/571
https://github.com/actions/cache/pull/489
https://github.com/actions/cache/pull/474
https://github.com/actions/cache/issues/795
https://github.com/actions/cache/issues/46
We are trying to consolidate all such requests and come up with a common solution. Your contribution will definitely help move this request forward. Please allow us some time to circle back on this PR while we evaluate the possible solutions.

Good luck!


fix: layer select state not update properly

Overview
This PR fix three issues:

When create a new layer by drop the layer in Earth Editor, it doesn't got auto selected (No indicator shown on map).

This issue also lead to when create a new infobox for this layer the Infobox popup doesn't show automaticlly.
The bug comes from the state update: selectedLayer happens before layers data update so it got undefined.
Why it works before is bc the setLayers with new created object will lead to anothor selectedLayer update which can get the latest layers data.
In order to implement the update after layers data changed I replace the useUpdate with an internal implemtation (which refers to the source code of useUpdate but expose the state) and use the state as the dependence of selectedLayer.

When unselect a layer the plugin api select doesn't emit (the plugin will not be noticed as unselect).

For select event undefined is also meaningful and also should emit event to notice plugin there's no selected object.

Plugin API event select cameramove emitted too much (not as expected i think).

The value of args (used as dependency of useEffect inside useEmit) is something like [camera] or [selectedLayer.id] which would be differect in each update and make the side effect trigger every time. For example if you add cameramove event from API then when you change any state eg. change scene background color you'll find the  event also emitted. Also if you add a select event and select a layer then when you move camera the select event keeps emitting.

What I've done

Replace useUpdate with a internal implement and use its state as selectedLayer's dependency.
Replace undefined to [undefined] in order to emit plugin api select event when unselect.
Add useMemo to cache the array like [camera] and [selectedLayer.id] to avoid unnecessary emit.

What I haven't done
How I tested

Create new layer from earth editor and create new infobox for it.
Select & unselect layer and watch the event got with plugin.
Add select mousemove event from plugin and watch the event emit.

Screenshot
Which point I want you to review particularly
Memo

And I tested it. This also fix the infobox issue


method errorsFor doesn't work for custom errorElement

Sometimes there's a grouping of checkboxes or radio buttons where the automatic placement of error messages doesn't quite place it properly.
In that case I've always been able to manually place the label:
<label id="element_id-error" class="error" for="element_id"></label>

And when showLabel is called, it will check to see if a error has already been placed with errorsFor, then will just add the errorsClass and message to the existing element.
Unfortunately when errorElement is used, then the errorsFor doesn't properly use it.  It currently is hard coded to search for:
selector = "label[for='" + name + "'], label[for='" + name + "'] *";

If that were switched to:
errorId = name.replace( /(\[|\])/g, "\\$1") + "-error",
selector = this.settings.errorElement + "#" + errorId + ", " + this.settings.errorElement + "#" + errorId + " *";

It would properly handle the errorElement.

It appears your issue/solution is similar to mine. :+1:  see option 2. in https://github.com/jzaefferer/jquery-validation/issues/1494. I hope they fix this.

Does seem so, though it's important to escape [ and ] with \ in case the name is like user[name]

@kmd1970 @paladox would be great if one of you could put together a pull request for this, including the suggested fix and a unit test. For the replace you should use the pattern used elsewhere in the code already:
.replace( /(:|\.|\[|\]|\$)/g, "\\$1")

Unfortunately I have never written a pull request before.

Might be a good opportunity to learn! GitHub has some good articles: https://help.github.com/articles/using-pull-requests/

I could do the pull request, though I'm not sure what you're using for your unit tests. Not quite sure where to put that.

The unit tests are written using QUnit, all relevant files are in the test folder. In this case, you could add something to https://github.com/jzaefferer/jquery-validation/blob/master/test/error-placement.js

I'm unable to reproduce this issue. Please provide a testpage that shows the issue. I can then reopen this.


Document and test supported versions of Go

I'm pondering adding a compatibility statement similar to the otel-go one.
https://github.com/open-telemetry/opentelemetry-go/tree/main?tab=readme-ov-file#compatibility
We don't really publish new releases with new Go versions though, and since this code is fully auto-generated, this may be a bit too much.

Once this is merged, I will make the compatibility-test check required.

Once this is merged, I will make the compatibility-test check required.

I think you can do it even before merging 😉

It would prevent any other PR from being merged before this one.

Their size has vastly reduced. They are not empty though.
Their content is retrieved from running go mod tidy.


Travis templates.

@devoncarew @sethladd

As part of this, we should switch from unittest: any to test: any.

Cleaned up.  @devoncarew : maybe another once over?
Thanks!

lgtm!


Code cleanup/issue 937 flatten

Summary
This PR "flattens" meaning, removes the arr, mat and scal directories in the Math library's primitives.
The related issue is #937, and the bash script used to do the majority of flattening implemented by @roualdes can be found here: https://gist.github.com/roualdes/ea72fc0fbcac05329e562a0e60ee0a1d
Tests
There were a few cases that didn't compile, but it was mostly trivially modifying the includes. There's one test case I'm not sure about that I want the user to be aware of, and that is ./runTests.py test/unit/math/prim/fun/digamma.cpp.
I've changed the test case to EXPECT_THROW, instead of EXPECT_PRED1, this is really the only thing that need be addressed.
Side Effects
Yeah, we're changing the whole directory structure.
Checklist

[x] Math issue #(937)

[x] Copyright holder: (Copyright 2018 Andre Zapico)
The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[x] the basic tests are passing

unit tests pass (to run, use: ./runTests.py test/unit)
header checks pass, (make test-headers)
docs build, (make doxygen)
code passes the built in C++ standards checks (make cpplint)

[x] the code is written in idiomatic C++ and changes are documented in the doxygen

[x] the new changes are tested

ping @syclik.
A few things to guide your eyes:

formatting issues, I'm hoping that the autoformatter will take  care of the extra whitespace.
more than one namespace declaration from the concatenation. The tests still compile. It's really only an aesthetic change, do we care?
The digamma test is what I'm worrying about. What is EXPECT_PRED1 vs EXPECT_THROW? I modified this so catches the error throws, couldn't find out quickly what this test case is doing though

I'll take a look. And I'll make sure to answer your questions too.

No rush. Is there something trivial I can do to change the include order of these ~=150 files?
This is something I overlooked in the refactor

Order: meta needs to stay the way it is. Everything else can and should be
sorted alphabetically.
On Mon, Mar 11, 2019 at 4:56 PM Andre Zapico<EMAIL_ADDRESS>wrote:

No rush. Is there something trivial I can do to change the include order
of these ~=150 files?
This is something I overlooked in the refactor
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/pull/1153#issuecomment-471726311, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F37bJO_5LdP8386n7pc1RlB9ytfwks5vVsLqgaJpZM4bpgm7
.

Also, it's ok to collect a few cleanup issues.
On Mon, Mar 11, 2019 at 5:15 PM Daniel Lee<EMAIL_ADDRESS>wrote:

Order: meta needs to stay the way it is. Everything else can and should be
sorted alphabetically.
On Mon, Mar 11, 2019 at 4:56 PM Andre Zapico<EMAIL_ADDRESS>wrote:

No rush. Is there something trivial I can do to change the include order
of these ~=150 files?
This is something I overlooked in the refactor
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/pull/1153#issuecomment-471726311, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F37bJO_5LdP8386n7pc1RlB9ytfwks5vVsLqgaJpZM4bpgm7
.

@drezap, I'm confused. It doesn't look like any of the existing files were removed?

I could probably address much of

more than one namespace declaration, and
the order of the includes

with some (more) bash code, if anyone knows an obvious way to identify which are the C system headers.  Do they all just start with the letter c?

It doesn't look like any of the existing files were removed

@syclik, all existing files are untouched in the hopes that this will help smooth out the transition to flattened directories.

@roualdes, thanks for letting me know it was intentional. Can you let me know how you're thinking of consolidating the files?
Unfortunately, this will have some side effects:

git history won't be preserved since all files all new. (a git mv or even a git rm followed by a git add would try to maintain some link between the new and old files)
we'll drift; this essentially means we won't be able to merge in PRs until the whole migration is done.

I'm curious what you think. I don't have many bright ideas at the moment. This was one of the things I was thinking about when I mentioned it'd be tedious and difficult.

I was previously thinking that we'd simultaneously maintain flat and not-flat directory structures, asking PRs to switch over as best we could before they are merged.  When we are satisfied with the success of the switch to the flat structure, then we'd simply delete the not-flat directories.  This thought is likely to be less smooth than I imagine.
Will git mv -f properly retain the history, where -f forces the move to the present files within the flattened directory structure?
@syclik, I don't understand your question about consolidating files.  My understanding is that the files within the flattened directory structure are already consolidated.  If what I've said above did not sufficiently address this question, would you please restate your question about consolidating files?

@roualdes sorry for the impreciseness. I meant how are we going to go from having the flat and non-flat structure to just having the flat structure. I think you addressed it and I agree that I think it's going to be pretty rough going this way, but I could be wrong. Maybe we can simulate walking a PR through this process to see if it makes sense?
I don't know enough about git to know if that'll track properly. I know if we had moved the first files over, it would track. Then as another commit squashed files, then we'd probably lose the history on those, but at least we'd track through the first set of changes.

I don't think dual structures will help because we don't want
people committing things against the old directory structures
once we've switched.  And the only way I see we could pull it
off is by duplicating every file with an include of the new
file location.
I think that at some point, all the existing pull requests
are just going to have to update to the new structure.

I agree with @syclik, as there's no point in maintaining a dual structure. I think we can go ahead and set a date for which all PR's must have the new flat structure (for prim), and then we can just delete the old structure in this PR. It's a lot easier to get rid of the old structure (just takes removing some directories and files). Once the date is set, we can just manually move the PRs that were between this flattening to and that date to the flat structure, and that covers everything.

You guys are right, there's no point in maintaining dual structures.
I'm still stuck on maintaining a file's git history.  This current PR does not maintain file history, as @syclik noticed.  Calling git mv on each file does retain each file's git history.  But I don't understand how to git mv files that are replicated in multiple directories.  For instance, how could one use git mv to combine prim/scal/fun/Phi.hpp and prim/mat/fun/Phi.hpp into prim/fun/Phi.hpp?

I'm still stuck on maintaining a file's git history.

@roualdes, please don't get over stuck here. I don't think it's possible to link all git histories together. I'd suggest picking one of the files to maintain history and then go with that. (Meaning, git mv one of the unflattened files to the flattened structure, git commit at that point with a meaningful message, then apply whatever changes to it on top.)
Does that make sense? Maybe there's a way to link everything together, but unless it's easy, I wouldn't worry about it.

That makes a lot of sense, because it best fits with my understanding of git mv.  Thanks, @syclik!
@drezap, let me know how you think it would be best to proceed, here.  My read is that we should modify my script to git mv any file that is not replicated across the directories mat, arr, and scal.  How can we minimize your effort that follows running this script?

@roualdes So let me make sure I understand the problem:
Some files were not moved properly, that is, for example, there are files in stan/math/prim/mat/fun/ that aren't in stan/math/prim/fun/?
I think it would be best to just write another script that takes all files, that weren't moved over yet, and just move them over? If we modify the original flatten.sh then I'll have to revert all the changes I just made.

@drezap, not quite. All files we're moved, but none we're moved so as to retain the git history. This was my bad.
I should be able to rewrite the script to maximize the number of files who's git history is retained, even if not every file's git history is retained.
Unfortunately, I don't see how we can keep from reverting all changes we both made.  Maybe we could put your changes into the script? Maybe we could start a new branch and cherry pick the commits across branches -- I'm not positive that this is possible, just thinking out loud. What do you think?

@roualdes not a big deal. I would just do what's easiest for you, and go ahead and revert the changes I've made. You're doing most of the heavy lifting here, anyway. Just let me know when you think you've figured out how to make the move to retain a git history.
Andre

@roualdes, I agree. There should be a script that can run on the code base. Since there have been merged PRs since this started, this would miss the migration of those changes, as far as I can tell.

I updated the gist, to retain as many file's git histories as I could, remove duplicated namespace declarations, and fix some (but certainly not all) of the cpplint issues.  I couldn't manage to reduce more of the cpplint issues with my script.
In an attempt to make @drezap's life easier, I carried forward all the commits previously made.
There's still some references to the not-flat structure floating around, e.g.
In file included from ./stan/math/rev/core/autodiffstackstorage.hpp:4:
./stan/math/memory/stack_alloc.hpp:7:10: fatal error: 'stan/math/prim/scal/meta/likely.hpp' file
      not found

These should be easy, but slow to uncover, but I have to get to class.
Let me know what you all think.  Thanks.

I'm going to close this, because there will be pushes where it's not going to pass tests and it's a waste of resources to keep running jenkins, but I'll re open before the merge.
@roualdes I remember why we were doing the dual structure before. Now that we're deleting the other files, the dependent rev/fwd/mix parts will not compile due the changed directory structure. I'm thinking we can just get prim together, have Daniel have a quick look to make sure everything's going smoothly, and we can just propagate the same logic to the other directories. No need to do a seperate PR for fwd, mix and rev.

@roualdes OK here's where I'm at. Every directory: err, fun, functor, prob, vectorize but  meta compiles and passes tests with the most recent commit, meta depends on some rev  and fwd stuff and that will compile with the flattening of those.
I don't see any reason why we should hold off on flattening the rest of the library (unless you want to do rev first, then mix, or some sequence). Is it too much to ask to just carry forward the same logic but starting in the rev fwd and mix directories?

@drezap, I've been trying to clean up the make cpplint issue "Found C system header after C++ system header" in an automated way, but have failed repeatedly.  So, I'm now ready to move on as you suggest.  We can just tack this issue on to our list of things to clean up later.
Re dual structure: I recall other reasons now, too.  It's significantly easier to test and re-test a script when the files to be copied have not disappeared.
I'll move on to rev, fwd, and mix.  Let's hope the same logic will carry forward.
Is this thread where you suggest we continue discussions, or elsewhere?

"Found C system header after C++ system header"

@roualdes if this helps: This issue is caused by C includes before stan/math includes, the error depict the problem. For example, I noticed the most common is that #include <cmath> will be at the top, but it should go after stan/math includes but before other C++ stuff. For example, you can pretty safely move all stan/math includes to the top, and here's an example of includes properly ordered:
#include <stan/math/prim/mat/fun/Eigen.hpp>
#include <stan/math/prim/mat/fun/divide.hpp>
#include <stan/math/prim/mat/fun/divide_columns.hpp>
#include <stan/math/prim/mat/fun/squared_distance.hpp>
#include <stan/math/prim/scal/err/check_not_nan.hpp>
#include <stan/math/prim/scal/err/check_positive.hpp>
#include <stan/math/prim/scal/err/check_positive_finite.hpp>
#include <stan/math/prim/scal/err/check_size_match.hpp>
#include <stan/math/prim/scal/fun/divide.hpp>
#include <stan/math/prim/scal/fun/exp.hpp>
#include <stan/math/prim/scal/fun/square.hpp>
#include <stan/math/prim/scal/meta/is_constant.hpp>
#include <stan/math/prim/scal/meta/return_type.hpp>
#include <cmath>
#include <vector>

I'll move on to rev, fwd, and mix. Let's hope the same logic will carry forward.

Cool, thanks. Appreciated. Let me know if I can do anything to help

Is this thread where you suggest we continue discussions, or elsewhere?

if you want there's this https://discourse.mc-stan.org/t/the-flattening-refactor/7932, but I'm good with either.

@drezap, your example did help, thanks.  This example was typical of the examples I developed to attempt to solve the problem.  Since we had similar examples, it dawned on me that the problem was in my logic, not specification of the problem.  It turns out a regex that differentiates the includes based on whether or not a period is included does not work, as this incorrectly reorders stan/math/prim/fun/Eigen.hpp.  Instead a regex based on a forward slash does work.
Take away: There's only 3 remaining make cpplint errors, all  "whitespace/line_length" errors.  The autoformatter should take care of these.
The down side to me finding this solution so late is that it's going to make a git revert mess of the git log.

@drezap, I reverted all of your changes, re-flattened prim so that the oder of includes does not upset make cpplint, and then re-reverted all of your changes.  Thanks for your patience here.
Next, I flattened fwd, rev, and mix with a slightly updated gist.  In fwd tests pass for functor, meta, and prob, but fun does not pass.  Nothing in rev nor mix passes, but I've only seen two errors for these directories.  Would you mind taking a look?

@roualdes yep I’ll take a look tonight. Thanks for doing that.
On Tuesday, March 19, 2019, Edward A. Roualdes<EMAIL_ADDRESS>wrote:

@drezap https://github.com/drezap, I reverted all of your changes,
re-flattened prim so that the oder of includes does not upset make cpplint,
and then re-reverted all of your changes. Thanks for your patience here.
Next, I flattened fwd, rev, and mix with a slightly updated gist
https://gist.github.com/roualdes/ea72fc0fbcac05329e562a0e60ee0a1d. In
fwd tests pass for functor, meta, and prob, but fun does not pass.
Nothing in rev nor mix passes, but I've only seen two errors for these
directories. Would you mind taking a look?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/pull/1153#issuecomment-474583243, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALHebGBCBynzNhox2y4J8h5D3k07QrkYks5vYVB3gaJpZM4bpgm7
.

--
Best,
Andre Zapico

I haven’t looked at the code yet, but based on what you’re saying, I think
we should take a step back and plan a little better.
There should be smaller chunks that can stay consistent and can be merged
without needing to update everything all at once.
If everything changes at once, the review process will take a long time. A
very long time. (I’m not saying it can’t work, but it’s not going to be
easy.)
On Tue, Mar 19, 2019 at 5:03 PM Edward A. Roualdes<EMAIL_ADDRESS>wrote:

@drezap https://github.com/drezap, I reverted all of your changes,
re-flattened prim so that the oder of includes does not upset make cpplint,
and then re-reverted all of your changes. Thanks for your patience here.
Next, I flattened fwd, rev, and mix with a slightly updated gist
https://gist.github.com/roualdes/ea72fc0fbcac05329e562a0e60ee0a1d. In
fwd tests pass for functor, meta, and prob, but fun does not pass.
Nothing in rev nor mix passes, but I've only seen two errors for these
directories. Would you mind taking a look?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/pull/1153#issuecomment-474583243, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ_F00r4mUUCUPQM95XoKz7C1UCT9IMks5vYVB2gaJpZM4bpgm7
.

@syclik so a few commits ago we had the double structure. Everything in prim is doing great, but the rest of the library breaks when we take the bottom out. Would you be happy if we left the double structure with prim and then prim's arr, scal, mat, merged prim? Or how would you suggest going about it? There's a lot of codependency. I'll put some thought into it

I should have said something about what remains of the dual structure.  Ever since we started retaining (as many as I could) files' git history, we can't maintain the dual structure.  Calling git mv, which does an actual mv, removes the source file from the not-flat side.
I kept around the not-flat structure this time because it's a (albeit inaccurate) record of which files' history we've lost; thought that might be informative.  It's an inaccurate record because git mv seems to only sometimes retain a file's history, and sometimes not.  It's unclear to me why even with git mv a file's history is only sometimes retained.
Moving forward in smaller chunks certainly has its appeal, but I've failed to produce a strategy that doesn't seem worse than flattening everything at once.  Now that my brain is out of the weeds of getting a script that I'm largely satisfied with, I'll give this moving in smaller chunks idea some more thought.

I'm digging through this code and a lot of this is actually fine. Most stuff in prim compiles and passes tests, there's just a few files that didn't get moved over. Any way we can do a simple ls -l *.hpp | wc -l to check to see all the files were moved over? With prim we're mostly OK, but I haven't dug through and done tests for everything.
I'm looking at rev right now. I can report on exactly what's happening, and may be that can guide the flattening of rev? Could be just the few files not moved to the new directory structure that are causing the issue, but after I take a look I'll give a better answer.
But most of this actually looks good...

@roualdes yeah I think this is moving forward with, at least making sure all files are moved over properly, (there's a few left over in prim and rev that haven't been moved, so let's make sure they all get there).
With that, I think we can actually keep going. The compiler in rev mode for some err stuff is throwing me something that's implying there's an include issue somewhere (which is what we would expect).
So would you mind just going through and make sure all files are properly moved? There's just a few files that aren't being moved over.
After that, with some work I think I can get this running.

@roualdes and @drezap: thanks! I'm glad you're tackling this. I knew it'd be troublesome and there are some big decisions to be made.
If you could do smaller chunks that are complete, checking the lines of code should roughly line up. With the large changes, it's just too many files to track.
Have you thought about moving just meta first? That seems like we could flatten that and then knock of folders at a time (test + source).

This was the first thing I did, I have a script with it somewhere, but I remember dependency issues, which lead to the idea of doing it at once... I mean if Edward figures it out I'm game for it. I could be doing something silly

@drezap, would you mind double checking about some of the files you suspect to have not copied over?  I'd guess they have been appended to the appropriate flattened file.  The files left behind in the not-flat structure are there to remind us which files have lost their git history.  But the code should be accounted for in the respective flattened file.
@syclik, which meta folder(s) are you suggesting we move first, prim/*/meta?  Or */scal/meta? Or */*/meta?
The main issue we face is when say prim/ files #include rev/ files, or some other combination.  I'm going to refer to this as cross folder dependencies.
I feel like we originally set out to flatten just prim by itself.  Then we hit inter-structure (flat versus not) cross folder dependencies.  We had files in the flat structure #includeing files from the not yet flattened structure.  The obvious solution was to flatten more, so we did.
Maybe we could flatten first just the folders that aren't inter-related, one folder at a time?  This would avoid the cross folder dependencies issue for a little while longer.

Can the tests all be refactored so that they call the top-level includes rather than specific header files?  Then, if all tests pass, I think we can trust the change without going through it file by file.  I don't think we want to do that even if it is broken down piece by piece.
We should consider moving to a setup where we don't even try to test the individual header files and instead just test our big recommended includes.  It's just too much of a burden to support individual header files given our include-order dependencies from template instantiations.
I also think those should be just primitive.hpp, rev.hpp, fwd.hpp, and mix.hpp.  The distinctions at the scalar/arr/matrix are scheduled for demolition as part of this flattening.
I know this is slightly off topic---I just thought changing some higher-level policy might make the changes here a lot easier to digest.

On Mar 21, 2019, at 10:06 PM, Daniel Lee<EMAIL_ADDRESS>wrote:
@roualdes and @drezap: thanks! I'm glad you're tackling this. I knew it'd be troublesome and there are some big decisions to be made.
If you could do smaller chunks that are complete, checking the lines of code should roughly line up. With the large changes, it's just too many files to track.
Have you thought about moving just meta first? That seems like we could flatten that and then knock of folders at a time (test + source).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

The files left behind in the not-flat structure are there to remind us which files have lost their git history

missed this comment, I understand now.

Can the tests all be refactored so that they call the top-level

I agree. This would massively simplify the refactor, and the review process.

To ensure my understanding of refactoring all tests so that they call the top-level, is the following interpretation correct?  We should change, for instance, #include <stan/math/prim/mat/fun/divide_columns.hpp> to #include <stan/math/prim.hpp>, correct? Or written more generally, with some painful pseudo-glob code, change #include <stan/math/{prim,fwd,rev,mix}/**/**/*.hpp> to #include <stan/math/{prim,fwd,rev,mix}.hpp>, right?
If we all agree about such a refactoring, then I think it would be best to adapt my flatten.sh script to automate as much of this as possible.  Now, this will require re-running my script, which means that I'd have to carry forward all of the post-script commits.  Two follow up questions.

Is there a better way to carry forward all post-script commits than continue to revert and re-revert previous commits?

I'd like to further adapted my script to call git commit after every git mv to increase the probability of retaining git histories.  Will anyone be offended by the number of commits this would create?

@roualdes

We should change, for instance, #include <stan/math/prim/mat/fun/divide_columns.hpp> to #include <stan/math/prim.hpp>, correct?

Yes. You'll end up with a few repeated lines and want to remove those.

written more generally,

probably not, we don't want to include a directory if it's not being used.

I'd like to further adapted my

I think having a more thorough git history is fine.

@drezap, thanks!  I can get started early next week.

we don't want to include a directory if it's not being used

Correct.  I misspoke.  I wanted to say we should follow a pattern, grep style not glob style, so as to only make this change within a specific file for the includes within that file.

@drezap, would you please take a look at the latest push, to help me determine why some tests are not passing?
Instead of reverting and re-reverting your previous edits, I incorporated them into the updated flatten script.

@roualdes sure thing, I'll try to give an update today.

Instead of reverting and re-reverting your previous edits, I incorporated them into the updated flatten script.

appreciated

@roualdes whoops my time estimate was too optimistic.
But yeah, just some re-declarations, etc. I just have a few more bugs to fix. Bob's suggestion massively simplified the refactor for us.
I would say it's safe to go ahead and start propagating the same logic to the rev, fwd and mix directories, when you get the chance.
I'll keep modifying scripts to make sure all tests compile, etc.

@drezap no worries.  Thanks for looking this over.
Let me know if you think of a way that I can change my bash script flatten to avoid the re-declarations.  Maybe even just point me towards an example.
All directories are already flattened.

thanks, I'll get to it soon


Call external API when Track changes

I have an automated streaming Setup, with Macros and so on, all running as a NodeJS server. Part of this setup is to display lower thirds with song information. I wanted to write something like this myself and almost got it to work when I found your project.
Would it be possible to send out the currently Playing Tracks (tracks where the player would have the white border in Rekordbox) via a REST API to some server? Might probably be extremely easy, would also do a PR if appreciated.

Yeah that's definitely possible, could just add a line to this method to make the callout to a defined server: https://github.com/erikrichardlarson/unbox/blob/5e783f511eb30e4385bfc57b5b656566e646a149/src/index.js#L516
If you don't want to run your own API server, I'm also planning to push an update for Patreon subscribers somewhat soon that will allow them to send / get data from a server with a key provided in the app.
Just let me know what works best for you and I can set you up.

@erikrichardlarson seems like the place to do it, however I am working on a Kuvo-Proxy right now, you you have any insights into that topic?

Oh nice work, I haven't looked into this much, but I definitely want to explore this further. And I'd also like to do this for Serato.  What are you using to inspect the requests here?

@erikrichardlarson check out this tool: https://docs.mitmproxy.org/stable/ it generates a proxy. You can also install an SSL Certificate generated by this proxy to allow HTTPS packet inspection. This is how I figured out which endpoints KUVO uses.
Later this can also be used to do the actual scripting, however I am woring on this part right now.
As soon as I have a working example, I'll post the repo here.

@erikrichardlarson check out https://github.com/kellertobias/kuvo-proxy

awesome work! I'll give this a try and see if I can extend to Serato as well.

@erikrichardlarson I have migrated my app to node/electron, checkout my repo. also you can check out a little demo: https://www.dropbox.com/s/7g4tj39p6rylwe7/kuvo-proxy.mp4?dl=0

Dude this looks great, does your proxy work with Beatport link?

Isn’t beatport link a music streaming service such as Spotify? If rekordbox
sends the data to kuvo, it will work.
erikrichardlarson @.***> schrieb am Mo. 5. Apr. 2021 um
22:47:

Dude this looks great, does your proxy work with Beatport link?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/erikrichardlarson/unbox/issues/13#issuecomment-813639950,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABU6ODR7PUMK7TLKGB6GBU3THIOUPANCNFSM4XYQCP3Q
.

--
mit freundlichen Grüßen
Tobias Keller

@erikrichardlarson if you want, you can contact me (my mail is on my webpage: kellertobi.de)

@erikrichardlarson I would add serato to the features of my app. do you have a running license and could gather some data?


Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[x] Tests pass
[ ] Appropriate changes to README are included in PR

I signed the Contributor License Agreement.


Improve no_priority mail preamble

Coverage increased (+0.009%) to 26.229% when pulling 4fdbcf202342ad3e75b38f0c96c16c5ba7f0fbe4 on calixteman:add_explanation into 4d76799074e1d32a138fdf65eb3feaea7ca5cb46 on mozilla:master.

@sylvestre you just need to r+.


In LinkML branch: determine section membership with slot group

preferably as an option, in addition to the current use of is_a parents
slot_group definition
Sujay and I will work on this
This is probably the entry point in main_linkml.js

bump: the linkml datastructure branch does group according to slot_groups now :-) , but it doesn't yet order either sections or columns according to rank.

I think the slot_groups have ranks as well. I'll dig something up now.

As noted on Slack, I missed that “dh_section” is a slot, and under that we have various “sub-classes” entities, which are referenced by slots’ “slot_group” field. I’ll revise section sorting code to take that into account now.

The latest push should fix this.  For sort, I just had to add a call to fetch slot_group specs from overall specification in order to get rank.

Yes, section and column layout looks great now.
Did you say you thought that column jumping would be restored in this push? I tried once in my usual setup and it didn't work for me yet.

Show section isn't working either

There's some odd behaviour with slots that are a subclass of one section but have "slot_group" some other section.  Fields in "MIxS core" seems to be problematic this way re. show section, and jumping to them.  Looking into it.

There's some odd behaviour with slots that are a subclass of one section but have "slot_group" some other section

Oh! I don't I intended that to happen. I will look into that later today.

Not your problem.  It turns out menu wasn't refreshing list of template sections or jump to fields.  Fixed but have to fix something else before pushing commit on that.

This should be fixed with latest commit.  I didn't catch it earlier because I wasn't doing template change when testing the show/jump functionality.

Showing and jumping work in Chrome, but not after loading data and validating

K.  likely validation causing javascript error that is knocking out other js functions. I will test on your sample data.

I think this is done. @sujaypatil96 may have some ideas for removing some section arrangement functionality that's replicated between the python can javascript components, but that could be another issue.


SAM-2865: Student response Pop-up with wrong format

Very minor suggestion, otherwise LGTM

Fixed, thanks for the suggestion.

Sorry, I didn't realize. Fixed.

Thanks @ramonchu2k!


Unable to Save QLoRA Finetuned Models

System Info / 系統信息
cuda version: 11.8
HuggingFace transformers library version: 4.35.2
SwissArmyTransformer version: 0.4.8
Python version: 3.10.13
Operating system: Ubuntu 22.04.3 LTS
Who can help? / 谁可以帮助到您？
@1049451037
Information / 问题信息

[ ] The official example scripts / 官方的示例脚本
[X] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程
It appears that the official script finetune_cogagent_demo.py supports QLoRA finetune CogAgent through SwissArmyTransformer library:
# Some lines are omitted here
elif args.use_qlora:
        model.add_mixin("lora", LoraMixin(args.num_layers, args.lora_rank, layer_range=args.layer_range, qlora=True), reinit=True)

When this is enabled, the vRAM usage decreased dramatically. However, I met multiple problem when trying to save trained checkpoints.
Problem 1
SwissArmyTransformer library assume all dense layers have bias. Here is a snippet from sat/model/finetune/lora2.py:
def merge_linear_lora(lin):
    if lin.original.weight.data.dtype is not torch.uint8:
        weight = lin.original.weight
        out_dim, in_dim = weight.shape
        new_lin = nn.Linear(in_dim, out_dim, dtype=weight.data.dtype, bias=lin.original.bias is not None)
    else:
        import bitsandbytes.functional as F

        # This line will cause error
        weight = F.dequantize_fp4(lin.original.weight.data, lin.original.weight.quant_state).to(lin.original.bias.data.dtype)

        out_dim, in_dim = weight.shape
        new_lin = HackLinearNF4(in_dim, out_dim, bias=lin.original.bias is not None)
        # The rest is omitted

The problem can be solved by rewrite that line to weight = F.dequantize_fp4(lin.original.weight.data, lin.original.weight.quant_state).
Problem 2:
It seems like that if you didn't enable training LoRA on the vision transformer part, this part of the finetune_demo.py script will throw an error:
    if args.use_lora:
        model.get_mixin("lora").merge_lora()

        # This line will thow an error.
        model.get_mixin("eva").vit_model.get_mixin("lora").merge_lora()

        args.use_lora = False
        args.save = "merged_lora_cogagent"
        from sat.training.model_io import save_checkpoint
        save_checkpoint(1, model, None, None, args)

My solution is to comment it out.
Problem 3:
After applying all changes mentioned above, this error occurred when saving merged checkpoint:
Traceback (most recent call last):
  File "/home/user/CogVLM/finetune_demo/finetune_cogagent_demo.py", line 303, in <module>
    save_checkpoint(1, model, None, None, args)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/training/model_io.py", line 117, in save_checkpoint
    save_ds_checkpoint(iteration, model, lr_scheduler, args)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/training/model_io.py", line 159, in save_ds_checkpoint
    save_ds_checkpoint_no_optim(model, args.save, str(iteration), client_state=sd)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/training/model_io.py", line 170, in save_ds_checkpoint_no_optim
    model._save_checkpoint(save_dir, tag, client_state=client_state)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 3209, in _save_checkpoint
    frozen_param_shapes=self._get_zero_frozen_param_attributes(self._get_param_shape_func)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 3266, in _get_zero_frozen_param_attributes
    raise ValueError(f"failed to find frozen {param} in named params")
ValueError: failed to find frozen Parameter containing:
(omitted due to being a big uint8 quantized tensor)

Problem 4:
After applying those changes mentioned above, when loading the checkpoint dumped during QLoRA training, another exception occurred:
NFO:sat:[RANK 0] global rank 0 is loading checkpoint /home/user/checkpoint/10000/mp_rank_00_model_states.pt
Traceback (most recent call last):
  File "/home/user/CogVLM/finetune_demo/evaluate_cogagent_demo.py", line 240, in <module>
    model, args = FineTuneTestCogAgentModel.from_pretrained(args.from_pretrained, args, overwrite_args={'model_parallel_size': args.model_parallel_size} if args.model_parallel_size != 1 else {})
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/model/base_model.py", line 215, in from_pretrained
    return cls.from_pretrained_base(name, args=args, home_path=home_path, url=url, prefix=prefix, build_only=build_only, overwrite_args=overwrite_args, **kwargs)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/model/base_model.py", line 209, in from_pretrained_base
    load_checkpoint(model, args, load_path=model_path, prefix=prefix)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/training/model_io.py", line 241, in load_checkpoint
    missing_keys, unexpected_keys = module.load_state_dict(sd['module'], strict=False)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2138, in load_state_dict
    load(self, state_dict)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2126, in load
    load(child, child_state_dict, child_prefix)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2126, in load
    load(child, child_state_dict, child_prefix)
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2126, in load
    load(child, child_state_dict, child_prefix)
  [Previous line repeated 3 more times]
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2120, in load
    module._load_from_state_dict(
  File "/home/user/miniconda3/envs/env1/lib/python3.10/site-packages/sat/model/finetune/lora2.py", line 30, in _load_from_state_dict
    self.weight.data.copy_(state_dict[prefix+'weight'])
RuntimeError: The size of tensor a (12288) must match the size of tensor b (25165824) at non-singleton dimension 0

(I found out that 25165824 = 12288 * 4096 / 2, this might be helpful when guessing what happened.)
====================
It might be possible to train the model without deepspeed, but the memory usage increase is very significant.
I really appreciate the high quality model you open-sourced to the community, and I hope that being able to train QLoRA will make it more approachable. Thank you for the hard work!
Expected behavior / 期待表现
The QLoRA should be merged into the model and then save normally.

maybe disscusion in #270


Words Detection Merged Bboxes

Bug description
Hello,
I hope you are having a good day. I am having a problem where multiple words are being predicted as 1 and since recongtion model is not trained on spaces, We get a lot of bad results. (please see attached screenshot)
I have tried db_resnet (both torch and TF), fast (both torch and TF).

Any solution is greatly appreciated.
Code snippet to reproduce the bug
def _init_if_needed():
    global crop_classifier, det_model, rec_model
    if not crop_classifier:
        crop_classifier = keras.models.load_model("crop_classifier/")
    if not det_model:
        det_model = detection_predictor(arch='db_resnet50', assume_straight_pages=True, pretrained=True)
        det_model.model.postprocessor.bin_thresh = 0.3
        det_model.model.postprocessor.box_thresh = 0.3
    if not rec_model:
        rec_model = recognition_predictor(arch='crnn_vgg16_bn', pretrained=True)

Error traceback
No error Message - NA

Environment
Conda ENV built with latest doctr commit.
Deep Learning backend

Hi @ArsalanYounus007 👋
Could you try to make the light gray text darker (https://answers.opencv.org/question/237967/how-to-darken-faintdim-gray-text/) ?

I apologize for the confusion, The image was created by me to see the results over the original image using pil's blend function.
This is the crop of original image.

Hi @ArsalanYounus007 :wave:
I tested both models pytorch (fast_base & db_resnet50) - main branch.
But looks like you don't use the ocr_predictor.
Could you please check the steps beetween again :)

You mean to say det_predictor + rec_predictor != ocr_predictor?
Let me check

@ArsalanYounus007 correct some steps in the middle are missing like the padding removal from the detection results .. i agree we should adjust this

I have used ocr_predictor and it gives better words.
Do we have option to call detection post processing steps manually? So I can keep det_predictor + rec_predictor approach?

Mh what you need to do is:
loc_preds = [list(det_out.values())[0] for det_out in det_preds]
# Rectify crops if aspect ratio
loc_preds = self._remove_padding(pages, loc_preds)  # type: ignore[arg-type]

remove_padding comes from:
https://github.com/mindee/doctr/blob/9c92f5cbd0bf64706672c29bf2d43117815c1794/doctr/models/predictor/base.py#L107
I will try to find a time slot next week to move it directly to the detection predictor

Thank you @felixdittrich92 for all the help. :)


removed the underline and solved the issue1615

Fixes Issue
Closes #1615
Changes proposed

Changes made:- removed the underline from the copyright text
i.e from Rupali Haldaiya and Open Source Community
Screenshots

Note to reviewers
Made Changes in components/Footer/Footer.tsx

I would strongly oppose this change, as underlining provides the impression that a text is a hyperlink. Removing the underline could confuse many users, and it would have a negative impact on the user experience.
@rupali-codes
Please share your opinion as well.

I would strongly oppose this change, as underlining provides the impression that a text is a hyperlink. Removing the underline could confuse many users, and it would have a negative impact on the user experience.
@rupali-codes Please share your opinion as well.

Ooh interesting point @Anmol-Baranwal

I have a suggestion : Instead of a standard underline, you can create custom underlines with various shapes or patterns.
if we add a transition it would give a different look i have tested it .

Screencast from 23-08-23 03:19:30 AM IST.webm
with small code base.
@Anmol-Baranwal @CBID2

I> I have a suggestion : Instead of a standard underline, you can create custom underlines with various shapes or patterns.

if we add a transition it would give a different look i have tested it with small code base.

Screencast from 23-08-23 03:19:30 AM IST.webm  watch the video
with small code base.
@Anmol-Baranwal @CBID2

I can't watch the video @subhadeepbanerjee2003. Is there another way you can embed it here?

@Anmol-Baranwal I have given directly from my files and I am not facing any problem. Which problem are you facing?

@CBID2 I have given directly from my files and I am not facing any problem. Which problem are you facing ? I can send it to your email

I downloaded it and can't view the video for some odd reason.

@CBID2 I have given directly from my files and I am not facing any problem. Which problem are you facing ? I can send it to your email

I downloaded it and can't view the video for some odd reason. Maybe you can screenshot each frame and explain each part?

@CBID2 ok sorry to hear that . My point was when any user hover to those part a underline will appear after that as a transition before that no underline will be there.

@CBID2 ok sorry to hear that . My point was when any user hover to those part a underline will appear after that as a transition before that no underline will be there.

I get that @subhadeepbanerjee2003. I'm just wondering if there's another way to use your video to highlight this idea. I'm having trouble visualizing it.

@CBID2 can you give me your email id ?

@CBID2 can you give me your email id ?

I don't feel comfortable doing that. Sorry

@CBID2 is there any other way ?

@CBID2 is there any other way ?

I managed to view the video. Anyway it looks promising

@CBID2 nice to hear that and I can work on that

@CBID2 can you give me your email id ?

I don't feel comfortable doing that. Sorry

You can directly tag me. @CBID2
If there is any problem.
@shubhsharma19
You can directly convert the video and upload it here. You can put screenshots, you can use extensions.
There are hundreds of solutions.

@Anmol-Baranwal thanks for pointing out

@CBID2

It will be something like this.

What do you think of the issue @Anmol-Baranwal

@CBID2

It will be something like this.

What do you think of the issue @Anmol-Baranwal

I don't believe this will enhance the user experience. In fact, it's worse than having no hyperlink, as relying on aesthetics isn't the solution for most users. Additionally, there are considerations for motor accessibility.
Feel free to ask @rupali-codes for her opinion. She might provide valuable insights.

@CBID2

It will be something like this.

What do you think of the issue @Anmol-Baranwal

I don't believe this will enhance the user experience. In fact, it's worse than having no hyperlink, as relying on aesthetics isn't the solution for most users. Additionally, there are considerations for motor accessibility.
Feel free to ask @rupali-codes for her opinion. She might provide valuable insights.

I was thinking this too. It also could also induce annoyance in the user.

@Anmol-Baranwal I think it would be unique but yes opinions can be difference for individuals.

@Anmol-Baranwal I think it would be unique but yes opinions can be difference for individuals.

I totally agree.
But you can ask good UX designers, and they will say the same thing.
If you want to learn more about UX laws: visit https://lawsofux.com/

I have a suggestion : Instead of a standard underline, you can create custom underlines with various shapes or patterns. if we add a transition it would give a different look i have tested it with small code base. Screencast from 23-08-23 03:19:30 AM IST.webm  watch the video with small code base.
@Anmol-Baranwal @CBID2

Great suggestion!

@Anmol-Baranwal thanks for your info

@shubhsharma19 thanks for your appreciation but there are some laws as mentioned by @Anmol-Baranwal so we should follow those.

@rupali-codes
What to do with this issue?

@rupali-codes What to do with this issue?

i didn't see the issue earlier, and i guess its better the way it is. Removing underline would confuse the users.

@rupali-codes
Closing this due to the mentioned reason.
@Sahilll15
There are open issues with label up for grabs. Feel free to comment there, and I will assign you the issue. I hope you understand why we are closing this.

I gained some knowledge through this conversation..no loss for me🙃


SEP-31: Always submit transaction object

Preview is available here:http://stellar-demo-wallet-client-pr312.previews.kube001.services.stellar-ops.com/


Update django version to resove a bug

we need to update Django in order to fix this issue (before merging the helpdesk branch):
https://github.com/django/django/pull/10733

Updated the requirements in the helpdesk branch