API calls that dispatch multiple actions for keeping API response data and other stuff (like metadata, etc.) in separate parts of the store? (With sample code)

I'm working on what is basically a CMS and has about ~30 or so different model classes that are rather inter-related. The easiest way to work with them is definitely with normalizr as suggested. Now, for my redux store object, I'm thinking the best way to set things up will be:
store = {
    // Any loaded entities from the API go here
    entities: {
        // merged directly from from normalizr `entities`
        modelA: {
            1: { // ...
            // ...
        },
        modelB: { // ...
            // ...
    },

    // Keep track of all other normal state for managing UI, pagination, etc...
    modelAListView: {
        page: 1,
        selected: [1, 2, 3, 4, 5] // the ID numbers of the objects currently in view
        // ...
    },
    modelBDetailView: {
        result: 17 // id number of modelB in view
    }
    // etc
}

So, my question is, in order to get this to happen with redux-api-middleware, I need a reducer for the state.entities (the apiReducer), and then my individual reducers as normal for all of the different views and such.
But then, I have to dispatch two separate actions to make sure that (A) the apiReducer receives updates whenever the API gets called, and (B) the appropriate model reducer receives updates when the API call involves that particular model.
I have worked out a solution to do this using redux-thunk, but I would really appreciate any feedback on this approach. So far it's working very nicely, and means my actual API calls are super simple to make from within my redux action creators. I would love to know if there is a better way anyone else has come up with!
So, first, here's my helper utility to make API requests with secondary action effects:
// redux/helpers/makeApiRequest.js

// set a default API root (or headers, credentials, etc.) so we don't need to type these everywhere
const apiRoot = '/api/v1';

export function makeApiRequest (options) {
    const {method = 'GET', path, query, schema, secondaryActionTypes, ...rest} = options;

    const endpoint = url.format({query, pathname: path.join(apiRoot, path)});

    let apiAction;

    // return a function that takes dispatch and add the `redux-thunk` middleware
    return dispatch => {
        if (Array.isArray(secondaryActionTypes) && secondaryActionTypes.length === 3) {

            // These are API hits that require a secondary update in a related reducer
            apiAction = {
                [CALL_API]: {
                    method, endpoint,
                    types: [{
                        type: secondaryActionTypes[0],
                        payload: () => dispatch({type: API_REQUEST}),
                    }, {
                        type: secondaryActionTypes[1],
                        payload: (action, state, res) => onSuccess(dispatch, res, schema),
                                                         // see helper function below
                    }, {
                        type: secondaryActionTypes[2],
                        payload: (action, state, res) => onFailure(dispatch, res),
                                                         // see helper function below
                    }],
                },
            };
        } else {

            // This is a normal `redux-api-middleware` type action for actions
            // that don't require updates specifically to the API entities reducer
            apiAction = {[CALL_API]: {method, endpoint, ...rest}};
        }

        return dispatch(apiAction);
    };
}

function onSuccess (dispatch, res, schema) {
    return getJSON(res)
        .then(json => {
            const data = normalize(json, schema);

            // Dispatch the API Action (will merge with `entities` branch of store)
            dispatch({
                type: API_SUCCESS,
                payload: {
                    entities: data.entities,
                },
            });

            // Payload for the secondary action type, will typically be merged into
            // a related model reducer somewhere else in the store
            return {
                result: data.result,
            };
        });
}

function onFailure (dispatch, res) {
    return getJSON(res)
        .then(json => {
            // Same as the default error action from `redux-api-middleware`
            const payload = new ApiError(res.status, res.statusText, json);

            // Send to the API reducer and return for the secondary reducer
            dispatch({type: API_FAILURE, payload});
            return payload;
        });
}

Now the next one is super simple, to update the entities branch of the store:
// redux/reducers/entities.js

const initialState = {
    modelA: {},
    modelB: {},
    // etc.
};

function reducer (state = initialState, action) {
    switch (action.type) {
        case API_REQUEST:
            // ...
        case API_SUCCESS:
            if (action.payload && action.payload.entities) {
                return Object.assign({}, state, action.payload.entities);
            }
            break;
        case API_FAILURE:
            // ...
        default: return state;
    }
}

export default reducer;

Now calling the API from any action is as easy as:
function getModelA (query) {
    const endpoint = 'model_a';

    return apiRequest({
        endpoint, query,
        schema: Schemas.MODEL_A,
        secondaryActionTypes: [MODEL_A_REQEST, MODEL_A_SUCCESS, MODEL_A_FAILURE],
    });
}

and I will have access to all of the data I need in reducers that handle both API actions and MODEL_A related actions.
Comments/feedback/suggestions? Thanks!

Hey this is really great!
I'm just getting my feet wet with this library, but I'm going to apply what you have here to my own project and see if there's anything I would change :)

Hey @anyong!
The initialState that you define in your reducer, does that play the same role combineReducers would? Is that where you're going with that?

Thanks!

The models are just the normalizr schemas that will be passed to normalize(), I just put them all in a file called models.js. The initialState set up with empty objects for each model simply means you won't get a bunch of cannot access property foo of undefined errors before data is loaded, and any map over the entities will just be an empty array.

Thanks for the quick reply!
Gotcha. Just so I understand, it looks like you're getting rid of having to use combineReducers because everything is handled within the api reducer, right?
The reason why I'm asking is because I have a series of reducers that I use with combineReducers:
export default combineReducers({
  router,
  form,
  users,
  vendors,

  // etc..
})

The approach you're taking is a bit different, because the API reducer will handle everything on its own and normalize is actually defining the store at a top level, right? Not nested within a reducer?
Does that make sense?
Thanks

The top-level entities reducer only takes care of the data. Any other information you need about your state should be coming from another reducer, in particular, things like which models (by ID) are currently displayed on the screen, errors, and that sort of thing. I've got something like:
{
  entities: {
    modelA: { 1: { .... }, 2: { ... } }
  },
  modelAListView: {
    active: [1, 2]
  }
}

Make sense?

If the entities reducer is the top level reducer that handles other reducers then yes this makes sense, otherwise I think I have to play around with it a bit more and then come back with questions if any arise.
Thanks for helping me out :)
Thanks,
Peter Piekarczyk
https://twitter.com/peterpme
http://github.com/ppiekarczyk
On Thu, Apr 21, 2016 at 11:19 AM anyong
<
mailto:anyong<EMAIL_ADDRESS>
wrote:

The top-level
entities
reducer
only
takes care of the data. Any other information you need about your state should be coming from another reducer, in particular, things like which models (by ID) are currently displayed on the screen, errors, and that sort of thing. I've got something like:
{ entities
:
{ modelA
:
{
1
:
{
...
. },
2
:
{
...
} } }, modelAListView
:
{ active
:
[
1
,
2
] } }
Make sense?
—
You are receiving this because you commented.
Reply to this email directly or
https://github.com/agraboso/redux-api-middleware/issues/65#issuecomment-212992741
https://github.com/agraboso/redux-api-middleware/issues/65#issuecomment-212992741

Hey @anyong
Just wanted to let you know this all makes sense. I took a look through the real-world example and could match the ideas.
Thank you for helping me out!

Hey @anyong ,
After playing around with this for a few days. Here are some pain points:
CRUD.
This technique seems to be working great when you only have to fetch stuff. When you try to do much more than that, it gets a little complicated, especially with normalizr.

Ok, sorry for the lack of context. Did a ton of refactoring on my end.
For anyone else interested, this can DEFINITELY be cleaned up, but I just needed something that works for now.
Background: Pure CRUD Api:
DELETE: returns no response body, 204
GET ALL: keyed, ie: customers: [{}]
UPDATE: no key, ie: {_id: xyz}
import {
  CALL_API,
  getJSON,
  ApiError
} from 'redux-api-middleware'
import {normalize} from 'normalizr'
import {omit, values} from 'lodash'

const HEADERS = {
  'Accept': 'application/json',
  'Content-Type': 'application/json'
}

const CREDENTIALS = 'same-origin'

export const API_REQUEST = 'API_REQUEST'
export const API_SUCCESS = 'API_SUCCESS'
export const API_FAILURE = 'API_FAILURE'

const API_ROOT = '/api'

export default function ({
  method = 'GET',
  body,
  headers = HEADERS,
  credentials = CREDENTIALS,
  url,
  actionTypes,
  schema,
  ...rest
}) {
  const endpoint = `${API_ROOT}/${url}`
  let apiAction

  const config = {}
  config.method = method
  config.endpoint = endpoint
  config.headers = headers
  config.credentials = credentials

  if (body) config.body = body

  return dispatch => {
    if (Array.isArray(actionTypes) && actionTypes.length === 3) {
      apiAction = {
        [CALL_API]: {
          ...config,
          types: [
            {
              type: actionTypes[0],
              payload: () => dispatch({type: API_REQUEST})
            },
            {
              type: actionTypes[1],
              payload: (action, state, res) => onSuccess(dispatch, res, schema, method, url, state)
            },
            {
              type: actionTypes[2],
              payload: (action, state, res) => onFailure(dispatch, res, method, url, state)
            }
          ]
        }
      }
    } else {
      apiAction = {
        [CALL_API]: {
          ...config,
          ...rest
        }
      }
    }

    return dispatch(apiAction)
  }
}

function handleNormalResponse (dispatch, res, schema, json) {
  let data
  if (Object.keys(json)[0] === schema.key) {
    data = normalize(json, {[schema.key]: schema.type})
  } else {
    data = normalize(json, schema.type)
  }

  let ids = {}

  if (typeof data.result === 'string') {
    ids = data.result
  } else {
    Object.keys(data.entities).map(entity => {
      ids[entity] = Object.keys(data.entities[entity]).map(id => id)
    })
  }

  dispatch({
    type: API_SUCCESS,
    payload: {
      entities: data.entities,
      ids
    }
  })

  return {
    ids
  }
}

function handleDeleteResponse (dispatch, schema, url, state) {
  const id = url.split('/').pop()
  const ids = state[schema.key].ids

  const items = state.entities[schema.key]
  const toKeep = ids.filter(key => key !== id)
  const itemsToKeep = omit(items, id)

  dispatch({
    type: API_SUCCESS,
    payload: {
      entities: {
        [schema.key]: itemsToKeep
      },
      ids: toKeep
    }
  })

  return {
    ids: toKeep,
    deleted: id
  }
}

function onSuccess (dispatch, res, schema, method, url, state) {
  return getJSON(res)
  .then(json => {
    switch (method) {
      case 'DELETE':
        return handleDeleteResponse(dispatch, schema, url, state)
      default:
        return handleNormalResponse(dispatch, res, schema, json)
    }
  })
}

function onFailure (dispatch, res, state, method) {
  return getJSON(res)
  .then(json => {
    const payload = new ApiError(res.status, res.statusText, json)

    if (payload.status === 401) {
      window.location = '/'
    }

    dispatch({
      type: API_FAILURE,
      payload
    })

    return payload
  })
}

@peterpme
Hello, I do not know how you use this module, you give an example?. Thank you

@janjon

create api/xhr.js file, paste this in there
within your async action creators, call api
create an api reducer that handles API_SUCCESS,FAILURE,REQUEST, etc.

:)

@peterpme WHATEVER_SUCCESS   Can not receive specific entities. How do I get entities, I state it really should be ids do. If I want to read how to entities。
Thank you

So this was an edge case I was dealing with. The server would sometimes respond with a keyed payload and sometimes it would return the payload:
customers: {
  name: 'customer 1'
 // ...
}

or
{
name: 'customer 1'
}

Normalizr didn't like that, so I tweaked it a bit.

@peterpme  thank you, Can you help solve this, please。 #91

hi @peterpme , Sorry, I go again. I have a question that I want to be like real-world as
function fetchStarred(login, nextPageUrl) {
  return {
    login,
    [CALL_API]: {
      types: [ STARRED_REQUEST, STARRED_SUCCESS, STARRED_FAILURE ],
      endpoint: nextPageUrl,
      schema: Schemas.REPO_ARRAY
    }
  }
}

But we would like to achieve this in this example
function fetchStarred(login, nextPageUrl) {
  return  api({
    actionTypes: [
      { type: SOUNDFILE_REQUEST, meta: { login: login } },
      { type: SOUNDFILE_SUCCESS, meta: { login: login } },
      { type: SOUNDFILE_FAILURE, meta: { login: login } }],
  })
}

mapActionToKey: action => action.meta.login,  //To use it like this

There is no better way。

you will have to write your own call-api middleware for that unless this middleware supports it. It's relatively easy and http://redux.js.org/ actually has one for you to use :)

@peterpme This means that it's also not possible to listen for a promise?

@Gamemaniak whether you write your own custom api middleware or use this one, you can return a promise. I'm not sure what you mean by listening for a promise


chore(package): update prettier to version 1.13.2

Closes #200

:tada: This PR is included in version 1.7.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Why Hessian can get by activation ($H = XX^T$) ？

I don't really understand, isn't the Hessian matrix a second-order derivative matrix, so how can it be obtained by multiplying the activation value matrix by its transpose ($H = XX^T$) ?

SparseGPT operates in a layer-wise manner, solving the corresponding layer-wise pruning problem, given by Equation (1) in the paper. This is essentially a masked linear squared error problem of which the Hessian is indeed $XX^T$; you are right that the whole network's Hessian includes second-order derivatives, but the SparseGPT algorithm does not require this Hessian.

SparseGPT operates in a layer-wise manner, solving the corresponding layer-wise pruning problem, given by Equation (1) in the paper. This is essentially a masked linear squared error problem of which the Hessian is indeed XXT; you are right that the whole network's Hessian includes second-order derivatives, but the SparseGPT algorithm does not require this Hessian.

SparseGPT operates in a layer-wise manner, solving the corresponding layer-wise pruning problem, given by Equation (1) in the paper. This is essentially a masked linear squared error problem of which the Hessian is indeed XXT; you are right that the whole network's Hessian includes second-order derivatives, but the SparseGPT algorithm does not require this Hessian.

Thanks for the answer, but why the Hessian for the masked linear squared error problem is $XX^T$ and how was this result derived?

@Beatlesso , check the least squares estimator here: https://global.oup.com/booksites/content/0199268010/samplesec3

The first order derivative of the squared loss function is as shown in equation 3.7, the second order derivative is as shown in equation 3.10 (which is essentially the same as XXT if you interchange the row and column dimension).

@Beatlesso , check the least squares estimator here: https://global.oup.com/booksites/content/0199268010/samplesec3

The first order derivative of the squared loss function is as shown in equation 3.7, the second order derivative is as shown in equation 3.10 (which is essentially the same as XXT if you interchange the row and column dimension).

This is very useful, thanks!


I keep getting a "no such file" error after sending a message

After sending a message, I get an error that a .json file is missing.
info login Logging in...
info login Redirected to https:\/\/www.facebook.com\/
info login Logged in
info login Request to reconnect
info login Done logging in.
node:events:342
      throw er; // Unhandled 'error' event
      ^

Error: ENOENT: no such file or directory, open './extracts/incomingAttachments/2021-05-29T15-03-44.927Z.json'
    at Object.openSync (node:fs:583:3)
    at Object.writeFileSync (node:fs:2144:35)
    at Object.parseDelta (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\ts-messenger-api\dist\lib\formatting\incomingMessageFormatters.js:23:18)
    at MqttClient.<anonymous> (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\ts-messenger-api\dist\lib\api.js:272:78)
    at MqttClient.emit (node:events:365:28)
    at Arguments.<anonymous> (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:1265:27)
    at Object.MqttClient.options.customHandleAcks (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:183:150)
    at MqttClient._handlePublish (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:1258:15)
    at MqttClient._handlePacket (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:410:12)
    at work (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:321:12)
Emitted 'error' event at:
    at MqttClient.<anonymous> (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\ts-messenger-api\dist\lib\api.js:275:55)
    at MqttClient.emit (node:events:365:28)
    [... lines matching original stack trace ...]
    at work (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:321:12)
    at Writable.writable._write (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\mqtt\lib\client.js:335:5)
    at doWrite (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\readable-stream\lib\_stream_writable.js:409:139)
    at writeOrBuffer (C:\Users\namehere\Desktop\Node\fbchatbot\node_modules\readable-stream\lib\_stream_writable.js:398:5) {
  errno: -4058,
  syscall: 'open',
  code: 'ENOENT',
  path: './extracts/incomingAttachments/2021-05-29T15-03-44.927Z.json'
}

EDIT: So I went ahead and created the folder /extracts/incomingAttachments/ and it stopped giving me the error. Am I supposed to do that?

Thanks for submitting this bug. We use the "extracts" folder to store HTTP or websocket responses for analyzing purposes, but it shouldn't be in the package.
It's fixed now. Try to install the latest version of API from NPM to avoid this problem.

That seemed to have fixed it, thanks!


remove --py36

Fixes #703.

Pull Request Test Coverage Report for Build 927

4 of 4 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 96.101%

Totals

Change from base Build 924:
0.1%

Covered Lines:
2982

Relevant Lines:
3103

💛  - Coveralls

Pull Request Test Coverage Report for Build 927

4 of 4 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 96.101%

Totals

Change from base Build 924:
0.1%

Covered Lines:
2982

Relevant Lines:
3103

💛  - Coveralls


Lazy Loading a module which has some syntax error will crash browser

This is a:

[ x ] Bug Report
[ ] Feature Request
[ ] General Query

My version of UI-Router is: (version)
v1.0.10
Bug Report
Lazy loading a module which has some syntax error will crash browser.
In lazyload.ts
function updateStateRegistry(result) {
        if (result && Array.isArray(result.states)) {
            result.states.forEach(function (state) { return transition.router.stateRegistry.register(state); });
        }
        return result;

if module has error, the result is 'undefined'  not Promise<LazyLoadResult>, browser will be freezed then crash!!!
updateStateRegistry method should check result is undefined or not.

I fix that like this
function updateStateRegistry(result) {
            if (!transition.router.stateRegistry.states[result]) {
                throw new Error(result + ' not loaded');
            }

            if (result && Array.isArray(result.states)) {
                result.states.forEach(function (state) {
                    return transition.router.stateRegistry.register(state);
                });
            }
            return result;
        }

By the way I use Angularjs 1.6.6 + ocLazyload. ocLazyload success load a module but it has error, so states will not be registry successfully. updateStateRegistry method should check it.

I fix that like this
function updateStateRegistry(result) {
            if (!transition.router.stateRegistry.states[result]) {
                throw new Error(result + ' not loaded');
            }

            if (result && Array.isArray(result.states)) {
                result.states.forEach(function (state) {
                    return transition.router.stateRegistry.register(state);
                });
            }
            return result;
        }

By the way I use Angularjs 1.6.6 + ocLazyload. ocLazyload success load a module but it has error, so states will not be registry successfully. updateStateRegistry method should check it.

I fix that like this
function updateStateRegistry(result) {
            if (!transition.router.stateRegistry.states[result]) {
                throw new Error(result + ' not loaded');
            }

            if (result && Array.isArray(result.states)) {
                result.states.forEach(function (state) {
                    return transition.router.stateRegistry.register(state);
                });
            }
            return result;
        }

By the way I use Angularjs 1.6.6 + ocLazyload. ocLazyload success load a module but it has error, so states will not be registry successfully. updateStateRegistry method should check it.

updateStateRegistry shouldn't be called if there's an error.  The value returned by ocLazyLoad should be a rejected promise which should then trigger the error handler:
    promise = lazyLoadFn['_promise'] =
        services.$q.when(lazyLoadFn(transition, state))
            .then(updateStateRegistry)
            .then(success, error);

Can you show how it fails in a plunker or stackblitz?

I use browser-sync to live reload angularjs app. If I have a unfinished route js file which has syntax error, the browser will reload then crash.

Closing because there is no reproduction provided. Provide some way that I can replicate this (preferably stackblitz or plunker) and we can re-open this.


Pop up control and icon change depend upon type

is there way to show popup control instead of tooltip and also can we change icon using ship type

I made each marker with its own color.I can post the changes

I made each marker with its own color.I can post the changes

@mara392 can you do that, please?


postinstall: allow write access to cellars in repository.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

If HOMEBREW_CELLAR (or HOMEBREW_PREFIX) are children of HOMEBREW_REPOSITORY it’s important to deny write to the repository and enable write to the Cellar/prefix afterwards CC @tdsmith

Thanks!
On Wed, Oct 12, 2016, 00:38 Mike McQuaid<EMAIL_ADDRESS>wrote:

Merged #1261 https://github.com/Homebrew/brew/pull/1261.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Homebrew/brew/pull/1261#event-820584848, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAKnQelP1ghFp45D66zc1T-NS4qM5f4Hks5qzI5mgaJpZM4KTYaQ
.


Revert some of commit 9676da78a

I'm looking at implementing trio-click into a real world application.  Is it safe to either revert this commit and put back the required items (incorrect import, lines 438-441 etc)
https://github.com/python-trio/trio-click/commit/9676da78a3db5c55f2ad49fda25f58d76c78878a
or would it be preferred to just provide a PR removing the print/gc() parts?

done for some time


Adjust Proposal status

Please adjust the status of the proposal to this flow:

Running is the state directly after the proposal is created
Counting is the state when FinalizeVotesLimit is < then participating accounts and we have to finalize multiple times
Rejected is the state when proposal has <= 50% in favour
Accepted is the state when proposal has > 50% in favour
Implemented is the state when a proposal is marked Implemented by the dao owner
Faulted is the state when the dao owner marks a running proposal as faulty

We need a  new extrinsic mark_implemented that can only be called by a dao owner, only when its status is Accepted.
The reasoning here is the following:
Lets say the proposal is "make the logo bigger on our website", then at some point the proposal is accepted. Then there is a phase where someone goes to the website and actually make the logo bigger. Afterwards the dao owner can update the Proposal to Implemented.

So:

rename Active to Running
you are ambiguous about Faulted/Faulty (diagram vs text)
add Implemented state and mark_implemented extrinsic

Yes. Your call wether faulty or faulted.


Added the unique data

I have added a unique data combining all the 4 datasets.

@JorgeMiguelGomes Please review my pr and check if any changes are to be done.

@apeksha235 hi, please include the code, that generates the file, in app.py.
Thank you

@JorgeMiguelGomes Sure


Retrying a message will duplicate the message in the message view

No content

Issue created by BlueBubbles Bot in the BlueBubbles commnity on behalf of @zach in #android-bugs.

The old message kinda acts like it's resent, but when you leave and re-enter the chat, they show as failed. They shouldn't even still be there


Usage for resource management?

More of a comment than a bug, but here's a use case I discovered playing around with async generators: resource management.
const fsp = require("fs-promise")

async function *open(file, opts) {
    const fd = await fsp.open(file, opts)
    try { yield fd } finally { await fsp.close(fd) }
}

for await (const fd of open("/path/to/file", {mode: "r+"})) {
    const bit = await fsp.read(fd)
    // do other things...
}

// descriptor automatically closed, so no resource leaks!

This seems correct to me, but a strange pun, and it would be unfortunate if it became common for things which would only yield once. Note that synchronous iterators permit the same. I think it'd be nice to look into a mechanism that's more targeted here for resource management, without mixing it with iteration.

@littledan I agree fully with that. It is a hack, and is most certainly counter-intuitive. I agree that resource management shouldn't be crossed with iteration like that, and should at least have a different syntax.

I was merely pointing out an interesting finding.

Yeah, this is interesting. Let's close this thread, as I don't think there's any particular action to take on this proposal.


⚠ Bofrost.lu has degraded performance

In 75e4b5e, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1513 ms

Resolved: Bofrost.lu performance has improved in c1a3611.


Requests and urllib3 using one test platform?

Basically the needs of the two libs as far as test environment is concerned are the same. It looks like there would be much benefit if requests could reuse test platform provided by urllib3.

@shazow have (very lightly) been discussing the idea of merging our projects in the mid-future

That's really interesting.

In the meantime, I'm fine with requests "trusting" that urllib3 is tested and reliable.

I was not implying it's not the case. I think it would be easier to have one test platform used by both libs. Taking into account that requests already embeds urllib3 there's no reason not to use its test platform if it supports all features requests' tests need.

On the other hand, it's kind of nice that we're not testing against the same webserver in the event that one of our's is buggy, at least there's some chance we'd catch the bug on the other layer. :P

@shazow +1

On the other hand, it's kind of nice that we're not testing against the same webserver

Am I missing something or is test platform in urllib3 going to have multiple backends? If so then both libs would in fact have been tested even better than now, right?

@piotr-dobrogost Not sure what you mean by multiple backends.
Originally we had a wsgiref simple server, then we moved to an evenlet wsgi app server, now we're moving to a Tornado wsgi server and a socket-handler based thread server.

Oh, well I thought you're adding support for Tornado not changing to Tornado.

All in the name of Python3 :)


Passion Project Peer Grading

[ ] Uses Frontend

[ ] Uses Backend

[ ] Use API

[ ] Agile

HOOK:
Points: 3.6-4.0
Reason:
KNOWLEDGE:
Points: 3.6-4.0
Reason:
VALUE:
Points: 0.6-1.0
Reason:
WOW FACTOR:
Reason:

[ x] Uses Frontend
[ x] Uses Backend
[ x] Use API
[x ]  Agile

HOOK: 3.8
Points: 3.6-4.0
Reason: Great hook, very interactive experience and very interesting webpage for someone who loves soccer and Fifa! I really liked how they asked many questions to me and my friend about FIFA and our interest in Soccer, and I really enjoyed the design and the pictures that they used in their webpage background.
KNOWLEDGE: 3.6
Points: 3.6-4.0
Reason: They showed me the data values for all of the players and I was able to understand how their webpage was able to function correctly. Unfortunately their wasn't connected so it wasn't able to fully display the card stats after pressing the "open a pack" button, but from what I saw the rest was running well, its just that there was a problem with it connecting.
VALUE: 0.9
Points: 0.6-1.0
Reason: The project is just for fun and it works well. I believe that people who like FIFA and soccer will enjoy this project because it is based on the stats of the fifa players, and the whole experience of opening a "pack" for a soccer player just like in the game
WOW FACTOR:
Reason: Great concept, I love soccer and playing FIFA so this webpage was awesome!
Total: (8.3/9) = 92.22%

Hook: 3.8. I really liked how it was an interactive experience, and i like fifa so i found it interesting
Knowledge: 3.75 It showed usage of both frontend and backend to a satisfactory level. Could've done a bit more though.
Value: 0.95. Great concept, and very fun experience.
Wow Factor: Yes. I love the fifa concept, webpage was so fun!!
Overall: 8.5/9


Implement Download Pages

Context
We have designs for how the download pages should look. Let's implement them.
Solution or next step
Implement the download pages.

Hello @davidsmejia, let's split this issue into more smaller and digestible features to make it easier to review.
Feature 1: My Dataset Process View (4 screen types)
Feature 2: My Dataset Advanced Option
( *Without modals - there is a tab menu for "Species View" and "Experiment View" that has modals for Samples table)
NOTE: As discussed previously, the modals(5 types) for the download pages, they will be handled separatly and will not be included in this issue.
Please review the mockup and let me know if any questions/suggestions, thank you!


add ThresholdStopper()

add new callback to stop optimization if the func value is lower than a given (by the user) threshold
linked to #999

Hello @GuillaumeSimo! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file skopt/callbacks.py:

Line 276:80: E501 line too long (81 > 79 characters)


Create Handlebars Helper Card - Step 6: Subscribe to the card

Tutorial URL: https://developers.sap.com/tutorials/cp-mobile-cards-handlebar.html
Step 6: Subscribe to the card
I've followed the steps in the tutorial and I was able to subscribe to the card. But I don't see 3 available cards. Was that done in another tutorial? In https://developers.sap.com/tutorials/cp-mobile-cards-web-push.html the was created using the Web IDE App.

@gregorwolf handlebar tutorial has been updated with some UI level changes, please try again, you should see top 3 (as per default query).

@thecodester i haven't heard back from OP. Assuming it has worked for him.
In case, you want to close this issue.


Add property to override underline styles of a focussed textfield

I feel like this is valuable as it is consistent with underlineStyle.
Thoughts on this feature?
Suggested by: https://github.com/callemall/material-ui/issues/1419

@hai-cea
imo a basic but necessary addition. please pull this mans code!

Thanks @zuren @kevtastic !


How can I visualize saliency of the Conv3D?

Hello guys,
Would I like know how to visualize the saliency of the Conv3D. Can Help me?
Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.
Thank you!

[x] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

[x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Use this library designed to work with keras models: https://github.com/raghakot/keras-vis


possible async eval

Exposes an async evaluation for pipelining graph construction with compute.
Used like so in MLX LM:
    y, prob = _step(y)

    while True:
        sync = mx.async_eval(y)
        next_out = _step(y)
        sync.wait()
        yield y.item(), prob
        y, prob = next_out

One line 45, after setting the unordered deps map, you should reverse size to save overhead:
std::unordered_map<std::uintptr_t, std::shared_future<void>> deps;
deps.reserve(outputs.size());

https://github.com/ml-explore/mlx/blob/a36a4db3b3f1ad4669ea698c0aad09a2b11e8d80/mlx/transforms.cpp#L272
Here on line 272, you could use emplace which is faster than insert as it doesn't copy to memory.
cotan_map.emplace(o.id(), astype(cotans[cotan_idx], o.dtype(), s));

In general this should be faster than most inserts.

https://github.com/ml-explore/mlx/blob/a36a4db3b3f1ad4669ea698c0aad09a2b11e8d80/mlx/transforms.cpp#L318
On line 318, I think you can reserve based on size of primals right after (before the loop).
vjps.reserve(primals_.size());

https://github.com/ml-explore/mlx/blob/a36a4db3b3f1ad4669ea698c0aad09a2b11e8d80/mlx/transforms.cpp#L322
move here might avoid unnecessary copying:
vjps.push_back(std::move(cotan_it->second));

@mustafaaljadery send a PR for some of these if you like? They aren't so related to this diff, so I won't include them but some of them could be good suggestions!

@mustafaaljadery send a PR for some of these if you like? They aren't so related to this PR, so I won't include them but some of them could be good suggestions!

Ah that's my bad. Totally misunderstood the PR, I thought it was optimizations to be made to transforms.cpp so I was going line by line trying to see if there were memory allocation optimizations.

100+ tps out of the box 🚀
python -m mlx_lm.generate --model mlx-community/NeuralBeagle14-7B-4bit-mlx --prompt "Write a quick sort in python" --temp 0.0 --max-tokens 256
Fetching 6 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 68200.07it/s]
==========
Prompt: <|im_start|>user
Write a quick sort in python<|im_end|>
<|im_start|>assistant

Here's a quick implementation of the Quick Sort algorithm in Python:

```python
def partition(array, low, high):
    """
    This function takes an array, a low index and a high index, and returns the index at which the array is partitioned.
    """
    pivot = array[high]  # Choose the rightmost element as the pivot
    i = low - 1  # Index of the smaller element

    for j in range(low, high):
        if array[j] <= pivot:
            i += 1
            array[i], array[j] = array[j], array[i]  # Swap elements

    array[i + 1], array[high] = array[high], array[i + 1]  # Swap pivot with the element at i + 1
    return i + 1

def quick_sort(array, low, high):
    """
    This function takes an array, a low index and a high index, and recursively sorts the array using the Quick Sort algorithm.
    """
    if low < high:
        #
==========
Prompt: 246.650 tokens-per-sec
Generation: 101.639 tokens-per-sec


Update llama.cpp submodule to 4bd0f93

I can confirm this now lets you run the DBRX models.
One thing to note is Ollama's default repeat penalty of 1.1 will really hurt DBRX-instruct if you add it to the available models.


UnicodeEncodeError on flake8

UnicodeEncodeError raised when Flake8 check files with unicode text.
Environment: python 2.7.13, flake8==3.2.1, teamcity-messages==1.21.
Sample file:
# coding: utf-8

SomeVa='fsdf ываыва'  # Non ascci symbols

Command for run sample: flake8  --teamcity=True --output=report.txt
Stacktrace:
Traceback (most recent call last):
  File ".../bin/flake8", line 11, in <module>
    sys.exit(main())
  File ".../lib/python2.7/site-packages/flake8/main/cli.py", line 16, in main
    app.run(argv)
  File ".../lib/python2.7/site-packages/flake8/main/application.py", line 322, in run
    self._run(argv)
  File ".../lib/python2.7/site-packages/flake8/main/application.py", line 308, in _run
    self.report_errors()
  File ".../lib/python2.7/site-packages/flake8/main/application.py", line 274, in report_errors
    results = self.file_checker_manager.report()
  File ".../lib/python2.7/site-packages/flake8/checker.py", line 305, in report
    results)
  File ".../lib/python2.7/site-packages/flake8/checker.py", line 215, in _handle_results
    physical_line=physical_line,
  File ".../lib/python2.7/site-packages/flake8/style_guide.py", line 284, in handle_error
    self.formatter.handle(error)
  File ".../lib/python2.7/site-packages/flake8/formatting/base.py", line 69, in handle
    self.write(line, source)
  File ".../lib/python2.7/site-packages/flake8/formatting/base.py", line 170, in write
    self._write(line)
  File ".../lib/python2.7/site-packages/flake8/formatting/base.py", line 152, in _write
    self.output_fd.write(output + self.newline)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 222-227: ordinal not in range(128)

What's interesting: flake8 --teamcity --output=report.txt works fine

Yes, but argument --output=report.txt is not processed, and report.txt is not created.
Another way for reproduce that exception:

flake8 --output=report.txt --teamcity  samplefile.py
export LANG=C && flake8 --teamcity  trash.py

I think, the reason is that flake doesn't expect unicode characters in error message.
Also this error is reproduced at latest flake8==3.3.0.


Provide Windows error message

Yes, it is already in README. I just thought it would be nice if the build system explicitly fails Windows builds.

Test package available [here].
Just 2 LOC, thanks to Zig's beautiful build system.
Just wanna mention #74

Sorry, I was kind of on a vacation. Anyways, it should be good now

Btw I saw your talk on Showtime, and I just want to say it's awesome :D

Should @panic or std.log.err message be used?
The difference lies in that @panic gives a stack trace pointing to the build script where @panic is invoked, whereas std.log.err + std.os.exit seems cleaner. Here are the screenshots:
@panic:

std.log.err + std.os.exit:

I personally prefer std.log.err + std.os.exit because the stack trace is not really adding any useful information to the already self-explanatory error message. I want the crash message to be beautiful, so I will convert this to a Draft PR temporarily before a decision is made.

I just committed the version with less chunky error message


No easy/No obvious way to get detailed logging information via Email

The most common use-case for MailHandlers (NativeMailerHandler and SwiftMailerHandler.php) is to send detailed information about errors/exceptions to developers.
Developers need context to solve errors, so they ask for a stacktrace and a useragent and maybe more.
The documentation doesn't offer an easy way to configure or implement this behaviour.
I understand that this can be achieved by providing a custom formatter, but I think this should be as easy as it can be.

I am in a need of implementing a new feature, what exactly would like to be created here?
just user agent? I could simply attach "HTTP_USER_AGENT" to the web processor.
Stack trace? it could be added with a new processor... is that what you mean?

Well I think this has been somewhat solved by the HtmlFormatter that you can combine with email handler now to get nicely formatted and clear emails.
Regarding configuration, https://github.com/theorchard/monolog-cascade now lets you do this a bit nicer than just raw monolog usage.

mmm so it can be closed...
any features needed to implement? not something to big or tiny...?

I guess it can be closed yes. Not many things to do ATM I think, apart from monolog 2.0 but I need some time to review open PRs before piling more up makes any sense.


Is this package still being maintained ?

Just curious if this package is still being maintained ?
Don't see a lot of movement.

Yes it is. The main efforts are going on the 2.0 version under development. 1.0 codebase is still actively updated with bugfixes.


kinesis-consumer: race condition on shard offset map

In the Kinesis consumer, we have a race condition on the offsets map (kinesis.offsets) almost in everywhere. For example in:

line 299
line 228
and more

From my point of view adding a lock to make it mutual exclusive is not a good idea because it has an impact on performance mainly line 299. A better solution might be in the way that processShard just work with the pointer of its offset, not whole offsets map structure.
Moreover, the current implementation of processShard and updateShards  doesn't handle resharding properly. For example, if a shard expired and get deleted (either splitShard or mergeShards) the goroutine(processShard) which was responsible for it won't be ended and try to read from a shard that doesn't exist anymore.

Mutexes in Go are very lightweight, so unless you don't measure a performance problem they are good to go. However - I agree with your idea to couple the offset with the function because avoiding a mutex always wins.

Passing the offset to the process function does not work well because the offsets still need to be gathered to be written to the offset file.
If the offsets are stored as local variables there is no way of querying them from outside. You would have to push the results to another function for storage which will again create a congestion point, i.e. you will need a mutex there.
So the best approach IMHO is to add an RWMutex here.

:+1:


Editing Namespace creates a new namespace

Rancher Version: master 5-26
Steps to Reproduce:

Setup Kubernetes environment
Add Namespace
Edit the new namespace's Name and save

Results: Create a whole new namespace
Expected: Should just change the Name and not create a new one?

No longer applicable in 1.4.


Improved data categories

Person:

Background Facts (biological, historical)
Beliefs and Values
Personality Traits
Important Life Experiences
Preferences
Interests and Hobbies
Career and Professional Life
Skills and Abilities
Relationships
Goals and Aspirations
Challenges and setbacks
Corporation
Mission & Vision
Principles/Values
Facts (+founding/history, -governance)

Personnel
Founding/History
Technology
Architecture

Governance

Organizational structure
Funding and financial information
Roadmap
Strategies

Programs and services
Partnerships and collaborations
Success Stories/Achievements
Membership

Get Involved
Audience/Beneficiaries

incorporated for MyLife organization through assistant's files, and for humans et al, evo-agent is the controller for this activity now


Change vegan and vegetarian emojis

Use 🌻 for vegan and 🌽 for vegetarian in correspondence with the icons used by the Studentenwerk.

Closed by 2ef3cc44bd72b63f7cef6f793aff2d6ba8b8daf7


Worker threads

Hi,
thanks for the lib, next allows us to profile node12 apps!
One question - I noticed job processed on worker_threads is not shown in the profiles. Do you know a way how to work around this?

Can you please include a repro if you can?

Sure:
const { Worker, isMainThread, parentPort } = require("worker_threads");
const fs = require("fs");

function work() {
  const fib = n => (n <= 1 ? 1 : fib(n - 1) + fib(n - 2));
  let x;
  for (let i = 0; i < 1e4; i++) {
    x = fib(1 + (i % 24));
  }
  if (parentPort) parentPort.postMessage(x);
}
const proftitle = "fibonacci";

if (isMainThread) {
  const v8Profiler = require("v8-profiler-next");
  v8Profiler.setSamplingInterval(100);
  v8Profiler.startProfiling(proftitle, true);
  new Worker(__filename).on("message", result => {
    const profiler = v8Profiler.stopProfiling(proftitle);
    profiler.export(function(error, result) {
      fs.writeFileSync("profile.cpuprofile", result);
      profiler.delete();
    });
    console.log(result);
  });
} else {
  work(); // worker thread
}

This is the profile:

In comparison this is how profile looks when profiling the fibonnaci calculation itself, without using worker_threads:

Flamecharts:

VS standalone fibonacci function, without worker threads:

I thought, well, if that doesn't work, I will just run the profiler inside the worker, right?  But when I try to run the profile inside the worker, it shouts Module did not self-register. at me.

Not sure if it helps in any way, but there was once this https://github.com/nodejs/node/issues/24016

I think it maybe v8 profiler can't work in work threads because isolate is designed not to be thread-safe. But when the work thread feature gets into Stability 2 (now is Stability 1: Experimental), the kernel may have a solution with it :)

@carera I have a repo that shows what is compatible with worker threads and what is not in terms of profiling. Bear in mind that it is still work in progress: https://github.com/slonka/pprof-nodejs-and-worker-threads/

Thanks @hyj1991 and @slonka . Do you by any chance have insight on whether the NodeJS team moves the worker threads into stability 2 and with what version?

@carera it should be in stable pretty soon: https://github.com/nodejs/node/issues/22940#issue-361580445

Hi all, so the workers feature is now flagged stable. What are the consequences?

https://github.com/nodejs/node/pull/31569

@carera  since can take heap snapshot from parent thread in this OR, I think the implementation for cpu profiling will come soon.

@hyj1991 I'm interested in profiling worker threads, and was wondering if there any updates to the state of affairs.
I had a look through the Node repo and couldn't find any issues/prs to add an equivalent CPU profiling API akin to the heap snapshot one, as it seems they're trying to cover that functionality off with the inspector module.
Do you think it would be possible for us to build worker thread profiling on top of the inspector profiling machinery it already has, or do we need something different added to Node?
I'm willing to help out implementing this but I'd need a bit of guidance.

Disclaimer: I did not work on profiling in node.js since sep 2019.
@Widdershin here is a working example of profiling inside worker threads using inspector module: https://github.com/slonka/nodejs-production-ready-profiling/blob/master/node-inspector.js

That's a great reference, thanks @slonka. Just curious, did you run into any issues with the inspector module at any point? My biggest reluctance is that stability banner.

Just curious, did you run into any issues with the inspector module at any point? My biggest reluctance is that stability banner.

I vaguely remember talking to some Node.js devs and them saying that it's the preferred method but I can see that the stability did not change: https://nodejs.org/api/inspector.html#inspector_inspector . I would open up an issue and ask them what's the recommended way to profile inside worker threads with an ability to start/stop on demand.

For what it's worth, v8-profiler-node8 seems to reliably profile inside of worker threads as well, with a slightly nicer API and feature set. I'm not sure that there are actually any issues with the node8 version in that regard, so I'll report back after more testing.

Ah, I see the problem now. v8-profiler-node8 works for profiling worker threads, but only if it hasn't first been loaded in the main thread.
Seems like we need to be context aware! https://github.com/schroffl/node-lzo/pull/11
I'm going to have a go at making v8-profiler-node8 context aware, hopefully the same can be done for this library.

I managed to make v8-profiler-node8 work across threads, should be totally viable to do the same here.
https://github.com/hyj1991/v8-profiler-node8/pull/15

npm i v8-profiler-node8@next --save now supports CPU profile in the worker threads.
Refs:

https://github.com/hyj1991/v8-profiler-node8/pull/15
https://github.com/hyj1991/v8-profiler-node8/pull/16
https://github.com/hyj1991/v8-profiler-node8/pull/17

I will patch it to v8-profiler-next soon.

Landed in https://github.com/hyj1991/v8-profiler-next/commit/8eea60851be85e444cf933f838eba5c7a7bcfc70, you can use<EMAIL_ADDRESS>to do CPU profiling in worker_threads.
Ref: https://github.com/hyj1991/v8-profiler-next/issues/37


Add Nick Khoury to Contributors list

@nick-khoury, I'm quite elated about your PR. I wanna evolve this project to addresses various problems faced by first-time contributors. I'd love to learn about your journey in open source community, the problems, pain points you had etc.
Could you explain how you felt when you went through the tutorial, made a PR and learned that I merged it?


VW MQB: Škoda Superb Mk3

Škoda Superb Mk3
Covers all variants of the third generation Superb (2015-2019?).

[x] added to README
[x] test route added to test_car_models
[x] route with openpilot: b2e9858e29db492b|2021-03-26--16-58-42

Thanks to community Superb owner johndoe (2018).

This should be G2G once the test route is uploaded to the CI bucket.


Fixed scorm file duplication issue

Fixes SCORM File Duplication Issue
This PR resolves the issue where multiple SCORM blocks within a single unit display the last added file in all blocks, instead of showing the correct SCORM file for each block.
Steps to Reproduce:

Add a valid SCORM file to a unit as a component (e.g., component 1).
Add another SCORM file as a separate component (e.g., component 2).
Add another SCORM file as a separate component (e.g., component 3).
Click on "Preview" and observe the behavior.

Expected Behavior:
Each block should display its respective SCORM file.
Actual Behavior:
All blocks display the last added SCORM file, replacing the previous ones.
JIRA: https://edlyio.atlassian.net/browse/EDLY-6550
Screenshot before Fix:

Screenshot after Fix:

Most people don't have access to our internal jira. I suggest we copy here the information from https://edlyio.atlassian.net/browse/EDLY-6550

Most people don't have access to our internal jira. I suggest we copy here (in the commit message) the information from https://edlyio.atlassian.net/browse/EDLY-6550

I have updated the PR description.

In the future, please try to add the information directly in the commit message. This helps people running git blame.


ZSE40 No Longer Working After Version 1.0.5 [bug]

I have a Zooz ZSE40 that was working great in version 1.0.5. After upgrading to 1.1.0 the device appears to be no longer working. The device show's offline in Home Assistant and is missing a device id in zwavejs2mqtt.
The device doesn't appear to have a device id anymore and doesn't seem to be setup however in the debug logs I can see the sensor is reporting data as node 11. I have include a device.json file from before and after.
node_11 json.zip
Here is also the debug info:
{   "id": 11,   "deviceId": null,   "manufacturer": null,   "manufacturerId": null,   "productType": null,   "productId": null,   "name": "OfficeMultiSensor",   "loc": "Office",   "values": [],   "groups": [],   "neighbors": [],   "ready": false,   "available": false,   "hassDevices": {},   "failed": false,   "lastActive": null,   "interviewCompleted": false,   "firmwareVersion": null,   "isBeaming": false,   "isSecure": false,   "keepAwake": false,   "maxBaudRate": null,   "isRouting": null,   "isFrequentListening": false,   "isListening": false,   "status": "Unknown",   "interviewStage": "ProtocolInfo",   "_name": "OfficeMultiSensor (Office)" }

I was able to resolve this by excluding the device and re-adding it. Thanks for this wonderful software!


Multiple slashes become one slash

1. What version of Caddy are you running (caddy -version)?
Caddy (untracked dev build)
From go get github.com/mholt/caddy/caddy.
On 5:30 PM (UTC) 16 December 2016.
2. What are you trying to do?
Confirm bug exists even on master branch of Caddy after discovering it on Caddy 0.9.3.
3.
On macOS 10.12.2, create the above Caddyfile in $GOPATH/bin, run ./caddy after cd $GOPATH/bin in the terminal, and go to http://localhost:2015/path/https://www.google.com in the browser.
5. What did you expect to see?
I expect to see GET /path/https://www.google.com HTTP/1.1 in $GOPATH/bin/access.log.
6. What did you see instead (give full error messages and/or log)?
I see GET /path/https:/www.google.com HTTP/1.1 in $GOPATH/bin/access.log.
7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?
Go through the above.

Have you tried escaping the URL ?

@abiosoft Yes, I have tried it from XMLHttpRequest. Python's SimpleHTTPServer works without escaping.

This is intentional; we call path.Clean() to help prevent directory traversal attacks. It removes extra forward slashes. What did the escaped request look like exactly?

@mholt I have tried permutations of the following and they all result in one slash:
http://localhost:2015/path%2Fhttps%3A%2F%2Fwww.google.com
http://localhost:2015/path/https%3A%2F%2Fwww.google.com
http://localhost:2015/path/https:%2F%2Fwww.google.com
http://localhost:2015/path/https:\/\/www.google.com

I have also tried a recent version of nginx (1.11.6), and it does not have this issue. Can you disable path.Clean() by default unless turned on by some configuration in Caddyfile, or remove it and use other ways to prevent directory traversal attacks? Thank you.

Can you disable path.Clean() by default

Sorry, no.
What we might be able to do is look into what exactly is causing even the query-encoded slashes to be replaced and see if we can somehow treat those as literals instead of path separators.

@mholt May I know your reason? Since directory traversal vulnerability has been addressed by nginx since 0.8.17 (http://nginx.org/en/security_advisories.html) without query encoding/decoding, why Caddy cannot do the same?

This might be related to https://github.com/mholt/caddy/issues/643, specifically https://github.com/mholt/caddy/commit/f31875dfdeb73db9dddf6a2a175a6b0b349d9b28. Perhaps there is a better way to approach this?

@mholt Here is what I just found out while getting my application to work in nginx. If proxy_pass is used with URI (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass), the equivalent of proxy + without in Caddy, same issue occurs where slashes will be merged. To remedy, merge_slashes off; has to be put into the server directive of its configuration file (http://nginx.org/en/docs/http/ngx_http_core_module.html#merge_slashes).

I think that we can introduce merge_slashes option. WDYT @mholt ?

Hmm, why would someone ever want the slashes to not be merged?

Sorry, no.
What we might be able to do is look into what exactly is causing even the query-encoded slashes to be replaced and see if we can somehow treat those as literals instead of path separators.

I was seriously thinking of using Caddy for all of my projects going forward, but this forced path cleanup alone will be a show stopper for me. The whole web archiving community, including big players like Internet Archive, use plain vanilla query URLs (without any URL encoding) for the sake of convenience (as the primary lookup key for web archives is URI itself). Here is an example:
http://web.archive.org/web/*/http://example.com/index.html
This means we cannot use Caddy as a reverse proxy server for many tools including:

OpenWayback
PyWb
InterPlanetary Wayback (ipwb
MemGator
Memento TimeGate
CarbonDate
OldWeb.today / Netcapsule

These are just a few of many tools that serve responses based of plain URLs in their path. Not supporting it is like failing to serve an entire ecosystem. I encountered this same issue while using Go's standard net/http library for writing MemGator. I asked about some workarounds at StackOverflow.
While I understand that cleaning up /./ and /../ is important to prevent directory traversal attack, I think removing multiple successive slashes is not preventing from any security risks that I am aware of (I may be wrong here). Hence, I would really encourage and request to provide an option to turn the double slash cleanup off (it doesn't have to be on by default, but should be possible when needed). When using Caddy as a reverse proxy or loadbalancer, backend servers can take care of those attacks.

@ibnesayeed Hmm, interesting use case.
When someone has some time, we should find out where the slash collapsing behavior is coming from -- maybe verify that it is in path.Clean() -- and then see if we can work around that somehow.
Does nginx and others manipulate the URL at all by collapsing slashes? I'm talking about in the general case, not necessarily when proxying or anything specific (since I do know that it can collapse slashes when proxying since paths have to be combined).

When someone has some time, we should find out where the slash collapsing behavior is coming from -- maybe verify that it is in path.Clean() -- and then see if we can work around that somehow.

Yes, it is indeed happening in path.Clean():

Replace multiple slashes with a single slash.
Eliminate each . path name element (the current directory).
Eliminate each inner .. path name element (the parent directory)
along with the non-.. element that precedes it.
Eliminate .. elements that begin a rooted path:
that is, replace "/.." by "/" at the beginning of a path..

I would further note that Caddy also uses filepath.Clean method. Clean method defined in both path and filepath packages are very similar though.

I have put together all the necessary bits that is used in the path cleaning on the playground to test and make changes.

Going forward, here are three possible approaches that I can think of:

provide a config option to disable path cleaning completely
implement a duplicate path cleaning method locally and conditionally (using a config option) ignore the slash collapsing
implement a proxy path cleaning method and whitelist certain situations by replacing occurrences of :// in the path to something like PROTOCOL_SEPARATOR/ then call the usual path.Clean() method and then replace the temporary mask back to it's original form.

The latter approach seems more reasonable to me as it is easier to implement and it does not relaxes the security guards in place. In our practical applications I only see those protocol prefixes having double slashes that needs to be preserved, so there is no point in whitelisting every other such occurrence. If the term PROTOCOL_SEPARATOR/ has the chance of collision then we can randomly generate a long string as mask each time we call path.Clean().

Merry Christmas @ibnesayeed :) Thanks for your feedback on this.
I generally try to avoid hard-coding edge cases like that into Caddy; perhaps we can write a wrapper function over path.Clean() that can replace // with some random string, do the path.Clean(), then replace the random strings again. This way, for any case where we don't want slash collapsing, we can call the wrapper function instead.
What use is there for path collapsing outside of proxying where we merge two paths?

Merry Christmas @ibnesayeed :)

Thanks @mholt and wishing you a very happy Xmas and new year.

I generally try to avoid hard-coding edge cases like that into Caddy; perhaps we can write a wrapper function over path.Clean() that can replace // with some random string, do the path.Clean(), then replace the random strings again. This way, for any case where we don't want slash collapsing, we can call the wrapper function instead.

This exactly what I meant by the "proxy method" in the third point. Further, in the post-note I mentioned that a random string can be used instead of a fixed one for masking double slashes.

What use is there for path collapsing outside of proxying where we merge two paths?

I have only seen legitimate double slashes being practically used in paths when a URL is part of the path and that too only near the protocol part. That's why I suggested that if we only mask the occurrences of :// then we will be sure that http://, https://, and ftp:// etc. will be preserved if they appear in the path.
The use cases I mentioned in the web archiving ecosystem is actually not the standard approach, instead it is for the sake of convenience. Most of the web archiving related services actually work based on URIs as parameters, hence, to avoid issues where client failed to encode the URI, they just started supporting non-encoded URIs in the path. For example, if you want to setup a service to provide thumbnails of any URI, you could have a service endpoint that responds to /thumbnails/{URI}. One would expect that {URI} would be url-encoded, but if there are no other arguments required then the service can relax the need of URL encoding and extracts anything that falls after /thumbnails/ as the URI to process and generate thumbnail of. This approach has some downsides, such as inability to pass query parameters, because they are now part of the supplied {URI}, so additional parameters either need to be provided by special request headers or adjusting the endpoint path scheme itself. For example if the desired dimension of the thumbnail is to be sullied by the client then the service endpoint could be something like /thumbnails/{width}/{height}/{URI}. The clear advantage of this non-standard approach is the ease that allows anyone to change manually edit desired URL in browser's address bar or curl command without the hassle of url-encoding. Good or bad, this is what is widely in practice in many such cases.

I wrote a rough code to illustrate how can it be done. Please find it in action with various test cases on the Go Playground.
import (
	"fmt"
	"path"
	"strings"
	"time"
)

func cleanMaskedPath(p, mask string) string {
	if mask != "" && strings.Index(p, mask) >= 0 {
		t := fmt.Sprintf("%d", time.Now().UnixNano())
		p = strings.Replace(p, mask, t, -1)
		p = path.Clean(p)
		return strings.Replace(p, t, mask, -1)
	}
	return path.Clean(p)
}

func main() {
	p := "/a/b/../././/c/http://example.com/foo//bar/../blah"
	fmt.Println(cleanMaskedPath(p, "://"))
	//=> "/a/c/http://example.com/foo/blah"
}

Basically replacing all the occurrences of path.Clean(p) in the code with cleanMaskedPath(p, "://") will do the trick. If more than one masks are to be supported, the code needs to be adjusted a bit, but the idea will be the same.

Here is a more generic format of variable length masking:
package main

import (
	"fmt"
	"math/rand"
	"path"
	"strings"
)

func randomString(n int) string {
	const letters = "abcdefghijklmnopqrstuvwxyz"
	b := make([]byte, n)
	for i := range b {
		b[i] = letters[rand.Int63()%int64(len(letters))]
	}
	return string(b)
}

func cleanMaskedPath(p string, mask ...string) string {
	t := ""
	mm := make(map[string]string)
	for _, m := range mask {
		if strings.Index(p, m) >= 0 {
			t = randomString(10)
			mm[m] = t
			p = strings.Replace(p, m, t, -1)
		}
	}
	p = path.Clean(p)
	for m, t := range mm {
		p = strings.Replace(p, t, m, -1)
	}
	return p
}

func main() {
	p := "/a/b/../././/c/http://example.com/foo//bar/../blah"

	// No masking
	fmt.Println(cleanMaskedPath(p))
	//=> "/a/c/http:/example.com/foo/blah"

	// Mask only protocol separators
	fmt.Println(cleanMaskedPath(p, "://"))
	//=> "/a/c/http://example.com/foo/blah"

	// Mask all repeated slashes
	fmt.Println(cleanMaskedPath(p, "//"))
	//=> "/a/.//c/http://example.com/blah"

	// Mask all parent directories
	fmt.Println(cleanMaskedPath(p, "/.."))
	//=> "/a/b/../c/http:/example.com/foo/bar/../blah"

	// Mask protocol separators and parent directories
	fmt.Println(cleanMaskedPath(p, "://", "/.."))
	//=> "/a/b/../c/http://example.com/foo/bar/../blah"

	// Mask everything
	fmt.Println(cleanMaskedPath(p, "//", "/..", "/."))
	//=> "/a/b/../././/c/http://example.com/foo//bar/../blah"
}

I am not very familiar with the Caddy code base. If someone can tell me where this should logically go then I can make a PR. Also, if it is desired to make the masking configurable, then we will need another intermediate function that calls cleanMaskedPath() with appropriate masks rather than directly adding masks everywhere in the code.

@ibnesayeed

This exactly what I meant by the "proxy method" in the third point. Further, in the post-note I mentioned that a random string can be used instead of a fixed one for masking double slashes.
...
That's why I suggested that if we only mask the occurrences of :// then we will be sure that http://, https://, and ftp:// etc. will be preserved if they appear in the path.

I'm suggesting that instead of masking :// in the path, we mask only // -- why limit it to just ://, if we now assume that collapsing extra path separators is an error? But I can see in your examples you do both. That's fine, but we need to decide what the default behavior will be.
Meanwhile, the sample implementations you've posted look pretty good. You've raised some really interesting questions I need more time to think about -- and of course the feedback of others is welcome here too -- such as:

Which masking method is faster, and is there a better/faster way? Some benchmarks would be good here.
Should we collapse all "//" in the path or just "://" (I'm leaning towards "//" and if it's not safe to do that by default, then it needs to be configurable, but I see no reason to single out "://" specifically.)
Where does this go in the code base? (I imagine it'll be in server.go around here.)

Which masking method is faster, and is there a better/faster way? Some benchmarks would be good here.

I don't see an opportunity of benchmarking here, because so far there is only one implementation. The first code that I wrote was nothing, but a proof of concept to illustrate what I meant (and perhaps you to meant the same thing). Later I just generalized that so that variable number of masks can be applied (zero or more). I tried to put some thoughts already to optimize the code. For example, I could have avoided if strings.Index(p, m) >= 0 {...} check and directly apply replacements for all the masks then apply unmasking for all after path.Clean() call (masks that wont match, wont be replaced and there will be no harm). However, I realized that in general cases, there will be very few requests that would need any replacement, so checking the presence would save us from generating a random string for replacement if the mask is not present in the path. Additionally, mm (mask map) is only populated with the masks that have any matches, this way I am reducing the number of loop when unmasking (if no matches found for any masks supplied, the unmasking loop will be skipped completely).
We can certainly improve the performance of random replacement string generator code (that was refactored in the latter implementation). Additionally, we can talk about the optimal size of the mask replacement string. Keeping it too short will cause collisions (both in the existing original path components as well as other mask replacements, if there were more than one masks applied at once). Keeping it too long will take longer to generate the strings. One clever way to keep the strings short while avoiding potential collisions would be to prefix the replacement string with fairly unique string that may not be very common in paths, such as two underscores __{RAND_STR}.

Should we collapse all "//" in the path or just "://" (I'm leaning towards "//" and if it's not safe to do that by default, then it needs to be configurable, but I see no reason to single out "://" specifically.)

My vote for default masking would go for the protocol masking (://) only. Because collapsing multiple slashes in paths is absolutely fine and it is part of the canonicalization that does not change the essence of the path. Paths //foo//bar, /foo/////bar, /foo/./bar, ///foo/././bar and //foo/.//./bar all mean the same thing, /foo/bar. However, due to the commonly misused style of passing URIs without encoding in the path, that specific occurrence of double slash after the protocol name is what needs be preserved from collapsing. I have not seen any other cases in practice (there might be some) where multiple occurrences slashes in the path are meaningful. Additionally, this very specific masking (only ://) will help preserving performance as it will cause less replacements (both masking and unmasking), hence less attempts to generate random string and less chances of collisions.

If it is supposed to be configurable then we can have named cases of path masking that can then be translated internally how it is actually converted to an argument.
func cleanMaskedPath(p string, mask ...string) string {
	/* Implemented before */
}

func cleanPath(p string) string {
	masks := []string
	switch config["path_cleaning_mode"] {
	case "preserve_protocol":
		masks[0] = "://"
	case "preserve_slashes":
		masks[0] = "//"
	case "preserve_dots":
		masks[0] = "/.."
		masks[1] = "/."
	case "preserve_all":
		masks[0] = "//"
		masks[0] = "/.."
		masks[1] = "/."
	case "clean_all":
		// Do nothing
	default:
		masks[0] = "://"
	}
	return cleanMaskedPath(p, masks...)
}

This is a pseudo code that illustrates how we can have an intermediate function cleanPath() that can read the config and set the masking accordingly, then calls the cleanMaskedPath() with those masks.
However, if one config takes care of all the "issues", then we don't have to introduce unnecessary configuration options just for the sake of flexibility, unless there is some compelling reason. In my opinion, currently only broken case is the one reported in the first post of this issue and that can be fixed by masking ://.
We can still keep the chain of intermediate functions, so that in future if more masking is desired, we can take care of it. If certain situations in future warrant a configuration option then we can easily implement that using this method. Otherwise introducing more configs degrade performance, inflate the code base, add the burden of maintaining documentation, answer more support questions, and add complexity for users.
For now, we can just go with something like this:
func cleanMaskedPath(p string, mask ...string) string {
	/* Implemented before */
}

func cleanPath(p string) string {
	/* No unnecessary config lookup and case checking */
	return cleanMaskedPath(p, "://")
}

And then replace all the existing calls to path.Clean() with cleanPath().

I didn't mean to criticize your implementation -- I think it's pretty good. As this is a critical path in Caddy I want to ensure it's as fast as possible (within reason).

One clever way to keep the strings short while avoiding potential collisions would be to prefix the replacement string with fairly unique string that may not be very common in paths, such as two underscores __{RAND_STR}.

Yes, I was going to suggest this same thing, or at least, something like "caddy{RAND_STR}_" as a mask.
Want to submit a pull request? It'd be good to start getting the code under review. :+1:

Paths //foo//bar, /foo/////bar, /foo/./bar, ///foo/././bar and //foo/.//./bar all mean the same thing, /foo/bar

Until it doesn't, to somebody's weird edge case, like yours with :// :wink:
I definitely agree with you for keeping it simple. Make a pull request when you have a chance and we'll get on it!

I didn't mean to criticize your implementation -- I think it's pretty good.

On the contrary, I think it's good to criticize implementations for the sake of improvements.

As this is a critical path in Caddy I want to ensure it's as fast as possible (within reason).

Oh, I thought you were talking about some comparative benchmarking. I will try to grab some data from some logs to see how much extra overhead it causes. Although, I think there will be many factors that will affect timings, such as how many instances in in the test set would require masking or what's the average length of paths in the test set.

Yes, I was going to suggest this same thing, or at least, something like "_caddy{RAND_STR}__" as a mask.

Perfect!

Want to submit a pull request? It'd be good to start getting the code under review. :+1:

I will try for that. However, I am still not sure why there are many places where path.Clean() is called directly, not just the server.go line you reference earlier.

Until it doesn't, to somebody's weird edge case, like yours with :// :wink:

If it ain't broke, don't fix it!

Where does this go in the code base? (I imagine it'll be in server.go around here.)

Following are the places where path.Clean() is called:

mime.go#L24
to.go#L24
fastcgi.go#L341
upstream.go#L409
context.go#L298
server.go#L354

So, I created a package with test cases and benchmarks. My Go skills are limited, so I might have done things in not very elegant way.
// pathcleaner.go

package pathcleaner

import (
	"math/rand"
	"path"
	"strings"
	"time"
)

// Most efficient method for random string generation: http://stackoverflow.com/a/31832326
const stringLenght = 10
const letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
const (
	letterIdxBits = 6
	letterIdxMask = 1<<letterIdxBits - 1
	letterIdxMax  = 63 / letterIdxBits
)

var src = rand.NewSource(time.Now().UnixNano())

func generateRandomString() string {
	b := make([]byte, stringLenght)
	for i, cache, remain := stringLenght-1, src.Int63(), letterIdxMax; i >= 0; {
		if remain == 0 {
			cache, remain = src.Int63(), letterIdxMax
		}
		if idx := int(cache & letterIdxMask); idx < len(letterBytes) {
			b[i] = letterBytes[idx]
			i--
		}
		cache >>= letterIdxBits
		remain--
	}
	return string(b)
}

func CleanMaskedPath(p string, mask ...string) string {
	t := ""
	mm := make(map[string]string)
	for _, m := range mask {
		if strings.Index(p, m) >= 0 {
			t = "/_caddy" + generateRandomString() + "__"
			mm[m] = t
			p = strings.Replace(p, m, t, -1)
		}
	}
	p = path.Clean(p)
	for m, t := range mm {
		p = strings.Replace(p, t, m, -1)
	}
	return p
}

func CleanPath(p string) string {
	return CleanMaskedPath(p, "://")
}

// pathcleaner_test.go

package pathcleaner

import (
	"path"
	"testing"
)

var paths = map[string]map[string]string{
	"/../a/b/../././/c": {
		"preserve_protocol": "/a/c",
		"preserve_slashes": "/a//c",
		"preserve_dots": "/../a/b/../././c",
		"clean_all": "/a/c",
	},
	"/path/https://www.google.com": {
		"preserve_protocol": "/path/https://www.google.com",
		"preserve_slashes": "/path/https://www.google.com",
		"preserve_dots": "/path/https:/www.google.com",
		"clean_all": "/path/https:/www.google.com",
	},
	"/a/b/../././/c/http://example.com/foo//bar/../blah": {
		"preserve_protocol": "/a/c/http://example.com/foo/blah",
		"preserve_slashes": "/a//c/http://example.com/foo/blah",
		"preserve_dots": "/a/b/../././c/http:/example.com/foo/bar/../blah",
		"clean_all": "/a/c/http:/example.com/foo/blah",
	},
}

func assertEqual(t *testing.T, exp, rcv string) {
	if exp != rcv {
		t.Errorf("\tExpected: %s\n\t\t\tRecieved: %s", exp, rcv)
	}
}

func TestPathClean(t *testing.T) {
	for p, v := range paths {
		assertEqual(t, v["clean_all"], path.Clean(p))
	}
}

func TestPreserveAll(t *testing.T) {
	for p, _ := range paths {
		assertEqual(t, p, CleanMaskedPath(p, "//", "/..", "/."))
	}
}

func TestPreserveProtocol(t *testing.T) {
	for p, v := range paths {
		assertEqual(t, v["preserve_protocol"], CleanMaskedPath(p, "://"))
	}
}

func TestPreserveSlashes(t *testing.T) {
	for p, v := range paths {
		assertEqual(t, v["preserve_slashes"], CleanMaskedPath(p, "//"))
	}
}

func TestPreserveDots(t *testing.T) {
	for p, v := range paths {
		assertEqual(t, v["preserve_dots"], CleanMaskedPath(p, "/..", "/."))
	}
}

func TestCleanAll(t *testing.T) {
	for p, v := range paths {
		assertEqual(t, v["clean_all"], CleanMaskedPath(p))
	}
}

func TestDefaultMask(t *testing.T) {
	for p, v := range paths {
		assertEqual(t, v["preserve_protocol"], CleanPath(p))
	}
}

func maskedBenchmarkRunner(b *testing.B, mask ...string) {
	for n := 0; n < b.N; n++ {
		for p, _ := range paths {
			CleanMaskedPath(p, mask...)
		}
	}
}

func BenchmarkPathClean(b *testing.B) {
	for n := 0; n < b.N; n++ {
		for p, _ := range paths {
			path.Clean(p)
		}
	}
}

func BenchmarkPreserveAll(b *testing.B) {
	maskedBenchmarkRunner(b, "//", "/..", "/.")
}

func BenchmarkPreserveProtocol(b *testing.B) {
	maskedBenchmarkRunner(b, "://")
}

func BenchmarkPreserveSlashes(b *testing.B) {
	maskedBenchmarkRunner(b, "//")
}

func BenchmarkPreserveDots(b *testing.B) {
	maskedBenchmarkRunner(b, "/..", "/.")
}

func BenchmarkCleanAll(b *testing.B) {
	maskedBenchmarkRunner(b)
}

func BenchmarkDefaultMask(b *testing.B) {
	for n := 0; n < b.N; n++ {
		for p, _ := range paths {
			CleanPath(p)
		}
	}
}

 go test -bench=. -v
=== RUN   TestPathClean
--- PASS: TestPathClean (0.00s)
=== RUN   TestPreserveAll
--- PASS: TestPreserveAll (0.00s)
=== RUN   TestPreserveProtocol
--- PASS: TestPreserveProtocol (0.00s)
=== RUN   TestPreserveSlashes
--- PASS: TestPreserveSlashes (0.00s)
=== RUN   TestPreserveDots
--- PASS: TestPreserveDots (0.00s)
=== RUN   TestCleanAll
--- PASS: TestCleanAll (0.00s)
=== RUN   TestDefaultMask
--- PASS: TestDefaultMask (0.00s)
PASS
BenchmarkPathClean-8       	 2000000	       769 ns/op
BenchmarkPreserveAll-8     	  200000	     10133 ns/op
BenchmarkPreserveProtocol-8	  500000	      3026 ns/op
BenchmarkPreserveSlashes-8 	  300000	      4306 ns/op
BenchmarkPreserveDots-8    	  200000	      6174 ns/op
BenchmarkCleanAll-8        	 1000000	      1063 ns/op
BenchmarkDefaultMask-8     	  500000	      3236 ns/op
ok  	pathcleaner	11.387s

Currently the tests and benchmarks were run on three different input paths, which in no way is the representative of a real web server hits. However, it gives the generic idea of what overhead this masking would cause. To evaluate various factors; 1) we can change the value of the stringLenght constant and see how it impacts the overhead, 2) run benchmarks on each of the three (or more) paths separately and see the effect of presence or absence of the masked substring in the path, and 3) simulate the benchmark on some real data from more than one web server logs (including those that use URIs in the path).

That's pretty good @ibnesayeed! Nice work, especially since you say you are "limited" in your Go skills.
Since the vast majority of URLs won't have any replacements at all, I suspect there's practically no performance impact, and those that are impacted have only a minimal effect.
Would you make a pull request with using this new function in server.go? I see that path.Clean() is called in several other spots -- thanks for finding them -- I think that most of those, or at least some of them, are probably not needed now that we do it down at server.go for every request. Maybe path.Clean() is needed if the URL path is being changed by a middleware, but path.Join should take care of things and do it properly.
Anyway, please make a pull request with the test and that function in place (just put it in the httpserver package, not a separate one). I'll be happy to do a review. :)

Anyway, please make a pull request with the test and that function in place (just put it in the httpserver package, not a separate one). I'll be happy to do a review. :)

PR #1317 is ready for review. I have excluded benchmarking functions, but can be added if you want to keep them in the repo. I have added necessary inline comments in the code to make it easier to maintain the code in future.


implement spiffs for html

the main minified html page should be embedded to spiffs when the code is flashed and the html should be served from there
ideally being able to upload a new html page on the fly would be preferable but not require

needs a bit of fiddling but that will come with the webserver stuff


Improve dev-experience around npm linking multiple packages

Right now, it's a fair bit of work to link/unlink a whole bunch of packages for enterprise testing. It would be nice to have some magic scripts around this.

Fixed with https://deephaven.atlassian.net/browse/DH-11593, no changes made in this repo


metrics exporter is crashing periodically

This was first observed when a heavier and more consistent load was applied to the Brigade2 dogfood cluster.
$ kubectl logs -f brigade-metrics-exporter-795fdb5446-s5ldx -n brigade-metrics
2021/07/27 18:57:18 Starting Brigade Metrics Exporter -- version v0.9.0-alpha.1 -- commit e605d87
2021/07/27 18:57:18 Server is listening without TLS on <IP_ADDRESS>:8080
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x86ff70]

goroutine 13 [running]:
main.(*metricsExporter).recordMetrics(0xc000180480)
	/src/exporter/metrics_exporter.go:132 +0xb10
main.(*metricsExporter).run(0xc000180480, 0x9f3b40, 0xc0000b8000)
	/src/exporter/metrics_exporter.go:88 +0x8d
created by main.main
	/src/exporter/main.go:35 +0x2c6

Since we discovered that this bug was caused by an older build of brigade-metrics and didn't happen again with the most updated version, should we close this issue?

Leave it open for now and I'll double-check yhe current status of the metrics exporter in the dogfood cluster as soon as I'm able.

Closing this as this has shown itself to no longer be an issue.


Check standardize_names.R

Names of the q52_1_scale, q52_2_scale, ..., q52_10_scale variables should not change; now they are getting changed to q52_loop_1_scale_1, q52_loop_2_scale_1, ..., q52_loop_10_scale_1; and are becoming q52_scale_1 in the qt3 data file

For all questions with variable names with the word "scale" in them, these questions do not need to be reshaped:
35,36,37,38,52,55,75,76,79a,80a and 81a


Using Store.Msg.processMultipleMessages to broadcast message

Hey guys!
Please, anyone knows how to use Store.Msg.processMultipleMessages?
I don't know if it does what I want.
I share a link like this: https://web.whatsapp.com/send?phone=00000000000 and now I got a huge list of chats. I would like to send  a broadcast.
I think that looping through all chats and send message is a bad idea.
Thanks

I think there is no way to do what you want. You need to send the message to each chat.


XCode Signing Interface Has Changed

The instructions are out-of-date with respect to the location of the signing settings in modern XCode (12.2)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 42599e6f-3abc-e5c3-fdb7-f59b68cf2b6e
Version Independent ID: 4f3b38e7-1ac0-569c-b1ba-13c0e2c4df3f
Content: Quickstart: Create a Unity iOS app - Azure Spatial Anchors
Content Source: articles/spatial-anchors/quickstarts/get-started-unity-ios.md
Service: azure-spatial-anchors
GitHub Login: @msftradford
Microsoft Alias: parkerra

opened AB#{31158563}

Addressed with MicrosoftDocs/azure-docs-pr#141720
#please-close


support pulling manifest without Docker-Content-Digest header

What is the problem you're trying to solve
Support pulling images from a registry which doesn't include Docker-Content-Digest header for manifests.
Describe the solution you'd like
Support pulling manifest without Docker-Content-Digest header
Additional context
Reported in https://github.com/genuinetools/img/issues/236

Already fixed by #3245

Should we bring #3245 back to release/1.2; seems like it could be perceived as a bug fix for working with a broader set of registry implementations?

I would only suggest pulling in such a large client change if the fix is requested by the CRI plugin (ping @Random-Liu)

@estesp @dmcgowan We haven't got issue report for this yet in either CRI plugin or Kubernetes.
But this does look like a bug to me, given that even the OCI distribution spec doesn't require it.


"Horia" is listed in the drop-down menu

Description:
When user writes "Horia" the location appears in the drop-down menu next to the comune and the county which is a part according to the law
Precondition:
The website is up an running.

Step 1
Write in the search bar "Horia"

Expected results
The location appears in the drop-down menu as "Comuna Horia, CONSTANȚA"


How to use?

Sorry, I am unable to make this work, can you please tell me what I am doing wrong?
use Gettext\GettextTranslator;

$t = new GettextTranslator();
$t->setLanguage('de_DE');
$t->loadDomain('messages', ROOT . '/Locale');

echo $t->gettext('Welcome to wishthis');

/Locale/de_DE/LC_MESSAGES/messages.po (created with Poedit)
msgid ""
msgstr ""
"Project-Id-Version: \n"
"POT-Creation-Date: 2022-03-21 09:58+0100\n"
"PO-Revision-Date: \n"
"Last-Translator: Jay Trees<EMAIL_ADDRESS>"Language-Team: \n"
"Language: de_DE\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"X-Generator: Poedit 3.0.1\n"
"X-Poedit-Basepath: ../../../src\n"
"X-Poedit-KeywordsList: __\n"
"X-Poedit-SearchPath-0: .\n"

#: functions/gettext.php:13 pages/home.php:22
msgid "Welcome to wishthis"
msgstr "Willkommen zu wishthis"

home.php

<?= _('Welcome to wishthis') ?>

What's wrong?
The text is not translated, the original is shown when I am excepting the German version.

To use the GettextTranslator, the translations file must be in .mo format.
The example in the README.md was wrong, sorry. I just fixed it.

Hey, thanks!
Unfortunately, after generating an .mo file I haven't noticed a difference yet.
In an attempt to get more insight I dumped the GettextTranslator but it's empty, do you have any more ideas for me? :/
use Gettext\GettextTranslator;

$t = new GettextTranslator();
$t->setLanguage('de_DE');
$t->loadDomain('messages', ROOT . '/Locale');

var_dump($t);

Is empty
object(Gettext\GettextTranslator)#4 (0) {
}

The GettextTranslator is just a wrapper of the native Gettext extension for PHP to provide the same interface as the regular translator. If you see the code you will notice that internally uses the gettext and ngettext functions of PHP. This is why it's empty: the translations in the .mo file are loaded by the gettext extension, not by this class.
Problems with the native gettext extension are very common (it depends on the locales installed in your operating system, if you change the mo file, you have to reload the server to clear the cache, etc). So maybe it's easy to use the regular translator (a bit less performance but more easy to use).


RIKEN FANTOM5, and various CSV files outputted from analysing clusters

Hi again, I have a question about inferring receptor-ligand interaction with RIKEN FANTOM5. Could you give a brief description as to how you carried that out? There are a lot of atlases and tools, e.g. TET and USCS genome browser, available on the RIKEN FANTOM5 site, but it's also not very clear in the paper which data was used to carry out this part of the analysis (e.g. did you use CSV files of initial clusters with Seurat?).
In addition, could you please describe what the edgeR.sig.csv, edgeR.design_matrix.csv, edgeR.DE.csv, and edgeR.csv do? I gather that these and the RNK files are the outputs of the in_clusters and between_clusters parts of the analysis, and the RNK files are analysed using GSEA.
Thank you very much for reading!

I am not sure about the RIKEN FANTOM5 part.
For comparison, edgeR.design_matrix.csv includes design matrix used in edgeR, edgeR.csv includes all genes while edgeR.sig.csv includes significantly differential expressed genes only.
You may look at the code.
https://github.com/shengqh/Hurley2022scRNA/blob/main/seurat_sct_celltype_rename_edgeR_betweenCluster_byCell/result/20220209_edgeR_betweenCluster_byCell.r

Hi erthrall, thanks for your question about RIKEN FANTOM5. Dr. Paula Hurley did the receptor-ligand analysis. I discussed with her and she said she described what she did in the Methods of our manuscript (PMID: 36229464):
"Receptor ligand interactions were inferred using receptor-ligands reported in RIKEN FANTOM5 database. Significantly increased DEGs in clusters 5, 6, and 11 (cancer cells) compared to cluster 12 (benign epithelial cells) were compared to ligands reported in RIKEN FANTOM5 database. The expression of corresponding receptors was assessed in cluster 20 cells."
If you need further clarifications, please feel free to email her<EMAIL_ADDRESS>Thanks,
Hong Yuen

Thank you so much, Hong!


There is no way to get all tweet replies.

Is your feature request related to a problem? Please describe.
From my understanding, there is no way to get all the replies of a tweet. It seems like the only way to get the replies of a tweet is to hit the /search/recent route and supply a query param conversation_id which is the tweet id.
Describe the solution you'd like
A function that returns all tweet replies.
Describe alternatives you've considered
It might be possible to use the v2 search function to solve this. The issue with this function is that it requires you supply it some string to search for, when in this case all we need to supply is the query param conversation_id
Additional context
N/A
Sorry if there is already a way to do this and I've missed it!

Hi,
Yeah an implementation with object-only parameter can be made for .v2.search. You can still give an empty string, it should work too.

Just double checked, empty string throws this error.
node_modules/twitter-api-v2/dist/client-mixins/request-handler.helper.js:99
        return new types_1.ApiResponseError(errorString, {
               ^
Error: Request failed with code 400 - Invalid Request: One or more parameters to your request was invalid

Available on 1.11.3 :)


Branch readme update contents

Fixes #21

Made a mistake by creating my branch-README-... from an existing branch: DG update... that's why there are two  commits here when there should be 1. I'm putting this comment here as a reminder to myself for the future.

Codecov Report

Merging #58 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #58   +/-   ##
=========================================
  Coverage     72.31%   72.31%
  Complexity      400      400
=========================================
  Files            70       70
  Lines          1228     1228
  Branches        124      124
=========================================
  Hits            888      888
  Misses          308      308
  Partials         32       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b072f6...bc2fea3. Read the comment docs.


Committer's Guide

Cleanup and minor updates to the old Committer's Guide on the wiki:
https://github.com/AnalyticalGraphicsInc/cesium/blob/commit/Documentation/Contributors/CommittersGuide/README.md

This is ready.


Correct way to reference a inserted child component?

I am writing a example like the official Tree example .
But I extend it to add a 'adding a sibling node' feature, My code is on jsfiddle.
Expected:

by clicking "[+sib]" after some node , I can insert a sibling node just below it.
after then I wanna change the .mode attribute of the newly added node to some other values, so I need get the reference of the newly added item component.

I got:
After new sibling node added, but using the same offset in $parent.$children cannot find the correct node which newly added. Instead by using console.log and vue-dev-tools  I found :

In the same tick , $parent.$children cannot found the newly added node.
In the Vue dev tools, I found the index of the newly added node always be the last element in the $parent.$children .

Since I really need it to do some extra work after creation, I am wondering which is the correct way to get the reference of the newly created element by index ? I mean the $index in the order of the view that can been seen by eyes.
Thanks in advance

Hello @li3p ie3p
Thank your for your interest in this project.
However, your issue is a usage/support question, and the issue tracker is reserved exclusively for bug reports and feature requests.
We encourage you to ask it on the forum  or on gitter.

Thank you for replying. But currently the forum only allow github account as Oauth , after login into it , I cannot get the confirm email , so I cannot post to forum.

Sorry to hear that, I assume you checked your spam folder?
We had to disable all other modes of registering because we had massive problems with spam bots, unfortunately.

@li3p make sure to set your GitHub email to public when you register (you can turn it back to private after you get the confirmation email)

@LinusBorg , @yyx990803  thank you guys. I turned my Github email to public and re-logined into forum, still cannot got confirmation email after click the link on the reminder tips. and I am sure I checked my spam folder . I am using gmail as my email.


Terminate Declaration via Cabinet FE

As an authorized patient after I signed a declaration I want to be able to terminate it.
In order to be able to do such I need to have button 'Terminate declaration' (spec) with pop-up approve window. After termination return to page with 'Declaration History'.

This issue was moved to edenlabllc/ehealth.web#13


Add Alembic for Database Version Control

Adds Alembic for database version control/migrations
Add documentation on how to use Alembic
Create initial Alembic migration script

:adequately-juiced:

What does this emoji mean

i like to think its self-evident


Explore wrapup

Source
Originally posted by @rperlin-ela in https://github.com/Language-Mapping/language-map/pull/127#issuecomment-723375179
Tasks
Bugs

[ ] Support commas in URLs, e.g. /Explore/Macrocommunity/Shi'a,%20Sunni.
[x] Correct Vatican City flag (see #130)

Panels/breadcrumbs

[x] Panel title bar 24/7, sans breadcrumbs in Home (and maybe Details)
[x] ...and return the ? help btn there once again and drop it from the bottom nav.
[x] and also move the i from the map ctrl btns as well.

Card lists

[x] can nix the word "Example/s", but maybe just add ellipsis as elsewhere, or ideally at least where there's more to show ideally yes, makes the most sense, and i gave it a shot but will have to call wishlist on this. so ... 24/7 for now, better than empty space. will keep an eye out for solution to reach the ideal. Added it to wishlist in #132
[x] Card lists: rm card header icons where too redundant, e.g. on every language card in /Explore/Language. Likewise under Language Family and Neighborhood, though not World Region or Country since the colors introduce some variation.

Size/Status

[x] Remove from Explore
[x] ...and make them not clickable in legend

Blurbs

[x] ...then update legend wording and only show if World Region is the symb type. Might as well include the "Show world map" toggle as well.
[ ] Add blurb in Explore panels to encourage users to view Data:

Yes, agreed, let's use similar language to what's in the table. Like "To view results in the map, filter columns in the Data table" or something like that?

Ross- i think my edits above cover what's left of what we discussed. feel free to add/correct if i missed any!

New members of the panel title bar party are getting pretty snug but I think I played it well with the transparent green shadow overlapping the overflowed crumbs:

I tried ellipsis instead of shadow and it was fine but only for the very very last depths. Sometimes the level above that was so long that the crumb would wrap, but I didn't want to pursue ellipsizing that one as that was another level of difficulty.
I think the transparent shadow looks better anyway, especially considering the final depth crumb is often the title of the current view already. Aka pretty redundant and expendable.
Obviously the home icon is not useful on the home view, but the consistency with the other views felt better than "blank empty crumb space on home but nowhere else".
I think the only outstanding thing I have left with crumbs is to show /Details/LanguageName instead of /Details/ID
The right side bar icons are pretty snug but no to the point where fat-fingerers would struggle. Ok I'm not one of them so let's get some chubby-digited clumsies up in here for testing. 🤙🏼