Black-Litterman model of expected returns

I'm interested in implementing the Black-Litterman model for expected returns (equation (1) in here). The model has a lot of moving parts but I think that it can all be contained in a simple function such as black_litterman_return(). If it's useful/interesting/desired, I can make a PR.

@schneiderfelipe
Hi, hope everything is going well. I actually implemented BL separately for myself, and it's not too difficult to do. I suppose I am more concerned about user-entered parameters. For example, what is a good way to allow users to specify their views? Should the function accept the BL inputs as matrices? or should we help users to construct them?
In any case, this is the implementation of the formula itself:
        tau_sigma_inv = np.linalg.inv(tau * sigma)
        omega_inv = np.diag(1 / np.diag(omega))  # efficient inverse of diagonal matrix
        p_omega_inv = P.T.dot(omega_inv)
        term1 = np.linalg.inv(tau_sigma_inv + p_omega_inv.dot(P))
        rets = term1.dot(p_omega_inv.dot(Q))

But the code for producing the inputs (particularly the omega matrix) is a little bit more tricky.
What do you think?

Hi @robertmartin8,
That's great! I was thinking in something similar to your solution:
def black_litterman_return(gamma, sigma, Q, omega, P=None, tau=0.01):
    # should we accept the implied returns gamma or a reference portfolio?
    # we might keep things flexible, checking for both and using the one provided

    omega_inv = np.diag(1 / np.diag(omega))
    if P is None:
        P = np.eye(omega_inv.shape[0])

    p_omega_inv = P.T @ omega_inv
    tau_sigma_inv = tau * np.linalg.inv(sigma)

    A = tau_sigma_inv + p_omega_inv @ P
    b = (tau_sigma_inv @ gamma + p_omega_inv @ Q)
    return np.linalg.solve(A, b)

A linear system is solved to avoid an extra matrix inversion, which I believe is more efficient (but I haven't tested yet!). Overall, only the sigma needs to be inverted.

I suppose I am more concerned about user-entered parameters. For example, what is a good way to allow users to specify their views? Should the function accept the BL inputs as matrices? or should we help users to construct them?

I think we should start with sensible defaults (like above for P and tau, what do you think about those default values?) and build up from that. The most basic here would be to accept no views at all by default (i.e. return the equilibrium return vector?) but receive the matrices provided by the user for now.

But the code for producing the inputs (particularly the omega matrix) is a little bit more tricky.

Since there's more than one way to do it, we could implement one or two after we know the code with the BL matrices works.
I'm particularly interested in Idzorek's method (section 3.3.2 of this) and methods involving valuation (I can't remember the source right now).
If there's a concern about poluting the interface or making the function too complex, a separate function might be used to build up the views using various methods.
(As a side question, when you use this, do you adjust the covariance matrix to reflect the BL model too? (equation (3) here).)

In your code, what is gamma? Haven't seen it in the literature.

A linear system is solved

Good spot!

I think we should start with sensible defaults (like above for P and tau, what do you think about those default values?)

Agreed for tau, but not sure it makes sense to set P to be None as default. I would expect users to supply views.

return the equilibrium return vector?

Specification of the prior should happen in a separate function, possibly with the default being no prior. One other concern is that constructing the BL prior requires market cap data.

As a side question, when you use this, do you adjust the covariance matrix to reflect the BL model too?

I've been experimenting with that but haven't come to any conclusions one way or another as to which is better. It's also an open question as to whether the output should be shrunk, or whether you should compute the shrinkage estimate of the sample cov matrix before using the BL "weighted average".

In your code, what is gamma? Haven't seen it in the literature.

Sorry for not mentioning it. It's the vector of implied returns (capital gamma).
We might go with asking the user for all the data (only giving a default value for tau?) and explicitly asking for the implied returns, so that it's their job to calculate the market portfolio and so on. Alternatively, another function could be used to calculate the implied returns from the market portfolio, as you said (which I think is the best idea actually).

I've been experimenting with that but haven't come to any conclusions one way or another as to which is better. It's also an open question as to whether the output should be shrunk, or whether you should compute the shrinkage estimate of the sample cov matrix before using the BL "weighted average".

I asked because I believe there would be room for a risk_models.black_litterman_cov:
# "cov_matrix" might be more meaningful than "sigma"
def black_litterman_cov(cov_matrix, omega, P, tau=0.01):
    omega_inv = np.diag(1 / np.diag(omega))
    p_omega_inv = P.T @ omega_inv
    tau_cov_matrix_inv = tau * np.linalg.inv(cov_matrix)

    A = tau_cov_matrix_inv + p_omega_inv @ P
    return cov_matrix + np.linalg.inv(A)

The idea and its proof can be seen here. The author says it's due to He and Litterman (2002), although I haven't read the original paper yet.
Using the function above, the user could provide any covariance matrix she likes, probably calculated using one of the functions in risk_models.
The drawback I see here is that there's a lot of repeated code among my proposed functions (black_litterman_return and black_litterman_cov). What do you think about making a class (such as CovarianceShrinkage)? A basic sketch follows:
class BlackLitterman:
    def __init__(self, implied_returns, cov_matrix, Q, omega, P, tau=0.01):
        ...
    def posterior_return(self):
        ...
    def posterior_cov(self):
        ...

It's the vector of implied returns (capital gamma).

In the papers I've read that's normally denoted by capital Pi?

The drawback I see here is that there's a lot of repeated code

This is actually quite a tricky problem from a design perspective. In the current code I'm using at work, I've written BL as a class and it works very well. But that structure doesn't really fit with the PyPortfolioOpt API. It seems there are two options:

We repeat a bit of code

We define a BlackLitterman class in expected_returns.py, as you've suggested, then just import it into risk_models.

Option 2 feels a bit unexpected, so I think it might just be best to repeat a little: "Simple is better than complex. Complex is better than complicated"
Here's another question that I'd like your input on. When it comes to implementing BL, should we refer to the variables by their symbols (P, Q, Pi, Sigma, Omega) – which seem to be almost universally consistent in the literature – or should we use what they represent, i.e picking_matrix, views_vector, implied_returns, cov_matrix, views_cov_matrix?

It's the vector of implied returns (capital gamma).

In the papers I've read that's normally denoted by capital Pi?

You're absolutely right, I mixed up the letters :sweat_smile:

Option 2 feels a bit unexpected, so I think it might just be best to repeat a little: "Simple is better than complex. Complex is better than complicated"

Fine, let's go simple then. Only two functions (expected_returns.black_litterman_return and risk_models.black_litterman_cov) with the BL model in general terms. Since there's more than one way of calculating the implied returns, views, etc., we can leave all that to the user.

Here's another question that I'd like your input on. When it comes to implementing BL, should we refer to the variables by their symbols (P, Q, Pi, Sigma, Omega) – which seem to be almost universally consistent in the literature – or should we use what they represent, i.e picking_matrix, views_vector, implied_returns, cov_matrix, views_cov_matrix?

If the user is required to provide the inputs, having terms as close as possible to the standard literature might make it easier for new users. Equations can be included, referenced and comented in the documentation, so that people know what is each term in the code.

Hi, great work on this package.
I was just curious about the progress made on implementing the BL model? If either of you are working on a branch/PR?
Also about the views, do we leave the creation of the view matrix up to the user, or is there any plans to create some helper functions to do this?

Hi @Busteren, essentially the code is just as given above.
Your question about the views is quite a tough one and is the main reason why I haven't implemented BL yet (apart from being generally busy).
It is not trivial for users to enter views - they have to know the precise specification and interpretation of the each row. But likewise, views are so flexible that it's hard to define useful helper methods.
I suppose one of the possible helper functions is to map absolute views to a views matrix (I've implemented this before). But even then, there'd have to be pretty comprehensive docs to make it usable

@Busteren @robertmartin8 I plan on doing a PR real soon (probably today or tomorrow). Sorry for the delay, time was really short lately.

@schneiderfelipe
I've been working to integrate the code from your PR #55. I've decided that BL won't be much use for the majority of users unless it's got many helper methods to format views properly, so I'm working on some functionality where a user can input (absolute) views and the function will figure out the picking matrix and views vector. Right now I'm hoping to accept either a pandas series or dict, e.g:
{ "AAPL": 0.20,
  "GOOG": 0.15
  "XOM":-0.3}

will result in a 3xN picking matrix and 3x1 views vector. I will also make sure the API allows for advanced users who want to input relative views and who know what they are doing.
However, with this in mind, it doesn't make sense to split BL between expected_returns and risk_models anymore. TL;DR, you were right and I was wrong :)
I'll be adapting your code into the new class (which I am also putting in a new module since it doesn't fit anywhere else).
On a side note, when I get round to writing the docs, the Black-Litterman page will attribute your GitHub profile – let me know if you would prefer this to be linkedin/email/webpage instead!

An example of this (with some working tests) has now been pushed to the dev branch. Tomorrow I'll work on adapting your BL covariance code, as well as implementing a few more methods. For example, BL can be used to allocate a portfolio directly (without passing through another optimiser) by combining the expected returns with a market-implied risk-aversion parameter.

@schneiderfelipe
I've just pushed the finished Black-Litterman code. If you have a moment, could you please take a look? There are a few examples in tests/test_black_litterman.py which demonstrate its usage:
test_bl_returns_no_prior()
test_bl_relative_views()
test_bl_returns_all_views()

That's awesome, thanks for the hard work @robertmartin8! I just proposed minor changes to the tests in PR #57 (minor code duplication).
Besides, I think the code and the concept are very good. Only a couple of things:

I believe the formula you're using for implied prior returns in market_implied_prior_returns is actually excess returns, which raises the question whether this should be added the risk-free rate. We must check if it would fit the rest of the API. If that's the case, a minor change should be set to bl_weights so that the risk-free rate is subtracted from returns there.

market_implied_risk_aversion is awesome and very very useful!

I would suggest naming absolute_views something like absolute_Q or abs_Q, or calling Q views. This would ensure that the user understands the relationship between the two.

I think both bl_returns and bl_cov should explicitly say in the docs that it's assumed that omega is diagonal.

I believe inversion can be avoided in bl_weights, which is probably more robust/efficient. Probably something like
  raw_weights = np.linalg.solve(risk_aversion * self.cov_matrix, self.posterior_rets)

I believe portfolio_performance should use the result from bl_cov instead of the prior covariance matrix (self.cov_matrix is currently used).

@schneiderfelipe Cheers for the review! Always very useful to have a second pair of eyes.

I believe the formula you're using for implied prior returns in market_implied_prior_returns is actually excess returns

I'll look through some of the papers to get to the bottom of this.

I would suggest naming absolute_views something like absolute_Q or abs_Q, or calling Q views.

This is a fair comment, but for now, I am inclined to leave it because I want to make the point that Q is the views vector (a mathematical object that BL requires) while absolute_views is a dictionary of views supplied by the user and is thus conceptually very different to Q. My feeling is that if I were to call it absolute_Q, it could cause some confusion as the user might expect it to have the same type as Q.

I think both bl_returns and bl_cov should explicitly say in the docs that it's assumed that omega is diagonal.

Good catch! Will do this.

I believe inversion can be avoided in bl_weights

As always, nice spot with the linear systems. I'll run some tests to make sure it works as expected.

I believe portfolio_performance should use the result from bl_cov

Agreed, that makes more sense!

You were right about the excess returns thing. It's a simple fix, just adding back the risk_free_rate. I'm leaving it to the user to ensure that time periods match. It fits the rest of the API because covariance matrices are invariant to linear shifts and the computation of the risk aversion coefficient uses the excess return.

I think both bl_returns and bl_cov should explicitly say in the docs that it's assumed that omega is diagonal.

I have added a comment to the docs, but I also added a mechanism to support a custom omega_inv matrix. During my summer internship, I was building BL portfolios and non-diagonal
uncertainty matrices were possible. So I'm exposing it as an attribute such that advanced users can write:
bl.omega_inv = np.linalg.inv(my_custom_uncertainty_matrix)
w = bl.bl_returns()

I want to add a few more bits and pieces to PyPortfolioOpt before the v0.5.0 release (unrelated to BL), so I will hopefully be in a position to fully merge by early next week.


Include Dyall's basis sets

This includes Dyall's primitive basis sets for the periodic table from the data on Zenodo
https://zenodo.org/records/7574629
There were two obvious issues in the data files which I fixed in basis_set_exchange-historical
dyall.acv4z

zr has only 8 f functions
nb has 5 g functions and 3 g functions

dyall.aae4z

cs has 13 not 1 f functions

Dyall has also published the contracted sets, but the problem is that they are at least two component. The primitive sets should already be useful for most users

@bennybp tests green and ready to go

A hefty addition. Thanks!
Too much to thoroughly check, but my spot checks didn't come up with any issues.


Unable to Render Beyond z17

When trying to render tiles beyond z17 node-mapnik just produces blue, featureless tiles.

This doesn't provide us much detail in order to help you with a problem that you might have, can you please provide us with more details and ideally a repeatable test case that we can use to reproduce any problems that there might be in the code.
https://github.com/mapnik/mapnik/wiki/A-perfect-testcase

In my JavaScript I have my "class":
var MAP = function() {
	function initialize() {
		mapnik.register_system_fonts();
		mapnik.register_default_fonts();
		mapnik.register_default_input_plugins();
	};

	this.tile = function(request, response, style, x, y, z) {
		var cwd = process.cwd();
		var style_path = cwd + configuration.map.path.style + style + ".xml";

		if ( ! fs.existsSync(style_path) || z < configuration.map.zoom.minimum || z > configuration.map.zoom.maximum ) {
			response.writeHead(404);
			response.end();

			return;
		}

		var path = cwd + configuration.map.path.cache;

		try {
			path += z + "/";
			if ( ! fs.existsSync(path) )
				fs.mkdirSync(path);

			path += y + "/";
			if ( ! fs.existsSync(path) )
				fs.mkdirSync(path);

			path += x + "/";
			if ( ! fs.existsSync(path) )
				fs.mkdirSync(path);

			path += style + ".png";
			if ( ! fs.existsSync(path) ) {
				var tile = new mapnik.Map(configuration.map.tile.size, configuration.map.tile.size);

				tile.loadSync(style_path, { strict: true });
				tile.extent = sphericalmercator.xyz_to_envelope(x, y, z, false);

				tile.renderFileSync(path, { format: "png" });
			}

			response.writeHead(200, { "Content-Type": "image/png" });
			response.end(fs.readFileSync(path));
		}
		catch ( exception ) {
			response.writeHead(404);
			response.end();
		}
	};

	initialize();
};

The various configuration object members are for the style (.xml from .mml/.mss), minimum/maximum desired zoom levels, and the cache path where generated tiles get cached.
This is what I get with my carto style sheet:
[z17]
[z18]
As far as I can tell the carto style sheet allows for zoom levels above 17 (zoom >= 17 in a number of places).

I would try logging the extent of the map and see what happens for zoom level 18.
tile.extent = sphericalmercator.xyz_to_envelope(x, y, z, false);

Also instead of providing a zip file could you provide the xml style you are using? Maybe just link it as a gist?

z18 is giving me [ NaN, NaN, NaN, NaN ] for each tile. Lower zoom levels (obviously) output the expected (example: [ -8238077.160463155, 4970547.0753284395, -8237771.412350015, 4970852.82344158 ]).
I have also Gist'ed my compiled carto style sheet.

The sphericalmercator.inc.js file is in this gist.

The problem was not in node-mapnik but in my sphericalmercator.inc.js file. It seems z18 was hard-coded in there at some point. Thank you so much for your help! I would not have known where to look without this conversation!
Closing issue.


Horizontal scroll start and end events fired multiple times; no scroll happening

First of all, thanks a lot for this package.
I'm having some trouble adding it to my Nuxt.js app. What I'm basically trying to do is scroll to a particular
 in an "infinite" width container, so that whenever I land on a particular ID (a composer), the list shows said composer within the viewport:

The code
<template>

      <nuxt-link :to="{ name: 'composers-id', params: { id: composer.id } }">
        <photo :composer="composer"></photo>

{{ composer.name }}

      </nuxt-link>

</template>

export default {
  // ...
  data () {
    return {
      options: {
        container: '#list',
        onStart: (element) => {
          console.log('start', element)
        },
        onDone: (element) => {
          console.log('done', element)
        },
        x: true,
        y: false
      }
    }
  },
  mounted () {
    this.manualScroll()
  },
  methods: {
    manualScroll () {
        // Should scroll to i.e. #composer-40
        this.$scrollTo('#composer-' + this.$route.params.id, 500, this.options)
    }
  }
}

The output

I'm clueless as to why this is happening. Any idea would be appreciated.

Can you try delaying the scroll? I think nextTick should work, but if not maybe a setTimeout
Can you also log inside the manualScroll method, and see if that fires multiple times?

@rigor789 Added a 3s timeout (setTimeout(this.manualScroll, 3000)) and a log to manualScroll(), this is the outcome:

The output is exactly the same with this.$nextTick. Any idea why this is happening?

So it looks like the mounted is fired multiple times - is this a component that has multiple instances? Maybe you need to move the $scrollTo calls to the parent component, or the root Vue instance (with a setTimeout to make sure the DOM has the correct nodes)

Ok, so I've moved the component to the parent and repetition is gone:

Still no scroll though.
This is the code (tried with setTimeout too but it seems everything loads with mounted + nextTick - https://vuejs.org/v2/api/#mounted):
  mounted () {
    this.$nextTick(function () {
      console.log('next tick')
      if (this.$route.params.id) {
        console.log('has id')
        this.manualScroll(this.$route.params.id)
      }
    })
  },
  methods: {
    manualScroll (id) {
      console.log('On manualScroll()')
      this.$scrollTo('#composer-' + id, 100, this.options)
    }
  }

Worth mentioning that I'm trying to scroll the horizontal panel alone, not the full page (which isn't scrollable in the x axis).

Can you show me the DOM structure from devtools? I'm mainly interested in #list since that is the container you are trying to scroll

Sure:

      <nuxt-link :to="{ name: 'composers-id', params: { id: composer.id } }">
        <photo :composer="composer"></photo>

{{ composer.name }}

      </nuxt-link>

My layout scrolls horizontally in the text container (where each composer's biography stands), and then we have that chronology bar (#list) which scrolls horizontally.
Hope that helps :)

Since we have two possible scrolls and the mouse wheel only scrolls vertically, could that be interfering with the functionality?

Can you try adding overflow to the #list id? I believe the scrollLeft property doesn't do anything if the overflow is not specified.

scrollLeft can be set to any integer value, however:

If the element can't be scrolled (e.g. it has no overflow), scrollLeft is set to 0.
If set to a value less than 0 (greater than 0 for right-to-left elements), scrollLeft is set to 0.
If set to a value greater than the maximum that the content can be scrolled, scrollLeft is set to the maximum.

from https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollLeft

Man ... It was my fault all along. That overflow thing made me look into the hierarchy of the code and turns out my #list ... was not an unique element but one list per group of composers was being set (by another developer) ... So of course we had multiple elements with the same ID and the whole thing broke.
It's working perfectly now, thanks a lot for your effort!
One suggestion though - we could be able to offset things by a percentage (of the viewport) instead of a static number of pixels.


Cannot use UserRecord.customClaims

I tried writing a script to get custom claims and had some trouble reading the server's response. Some observations:

According to the Firebase docs, the custom claims are stored as a simple JavaScript Object.
According to this package's docs, UserRecord.customClaims is a dynamic.
UserRecord.customClaims.runtimeType returns JSObject. Note the capital S. This spelling doesn't match any other class I could find.
Trying to cast it to a JsObject from dart:js throws: CastError: Instance of 'PlainJavaScriptObject': type 'PlainJavaScriptObject' is not a subtype of type 'JsObject'. Note that here it refers to the same object as PlainJavaScriptObject whereas claims.runtimeType returned JSObject.
Modifying the code outputted by dart2js allows me to use it as a standard JS object.

The last one helped me pin down the issue -- the claims were never converted to a Dart object, so any attempt to use it as such will fail. To fix this I used dartify from the node_interop library:
final Map<String, dynamic> dartClaims = dartify(claims);

That works. I would've submitted a PR with this change, but I'm not quite sure where to put this function call. The UserRecord is a JavaScript binding with only external methods. Does this mean extending UserRecord and changing customClaims to a getter that converts it under the hood? I'm not quite sure what approach would fit in the pattern used in this package.

I came here to open the exact same issue. thanks for the temporary solution


Uncaught Error: ENOENT: no such file or directory, stat ''

[Enter steps to reproduce below:]
This occurs on showing the tooltip for my .hs files.
Atom Version: 1.3.3
System: Mac OS X 10.11.2
Thrown From: haskell-ghc-mod package, v1.6.0
Stack Trace
Uncaught Error: ENOENT: no such file or directory, stat ''
At ATOM_SHELL_ASAR.js:0

Error: ENOENT: no such file or directory, stat ''
    at Error (native)
    at fs.statSync (fs.js:854:18)
    at Object.fs.statSync (ATOM_SHELL_ASAR.js:225:16)
    at Object.module.exports.Util.isDirectory (/Users/hn/.atom/packages/haskell-ghc-mod/node_modules/atom-haskell-utils/lib/util.js:22:21)
    at GhcModiProcessReal.module.exports.GhcModiProcessReal.getRootDir (/Users/hn/.atom/packages/haskell-ghc-mod/lib/ghc-mod/ghc-modi-process-real.coffee:31:23)
    at GhcModiProcess.module.exports.GhcModiProcess.getRootDir (/Users/hn/.atom/packages/haskell-ghc-mod/lib/ghc-mod/ghc-modi-process.coffee:308:5)
    at GhcModiProcess.module.exports.GhcModiProcess.queueCmd (/Users/hn/.atom/packages/haskell-ghc-mod/lib/ghc-mod/ghc-modi-process.coffee:144:21)
    at GhcModiProcess.queueCmd (/Users/hn/.atom/packages/haskell-ghc-mod/lib/ghc-mod/ghc-modi-process.coffee:1:1)
    at GhcModiProcess.module.exports.GhcModiProcess.getInfoInBuffer (/Users/hn/.atom/packages/haskell-ghc-mod/lib/ghc-mod/ghc-modi-process.coffee:216:6)
    at GhcModiProcess.getInfoInBuffer (/Users/hn/.atom/packages/haskell-ghc-mod/lib/ghc-mod/ghc-modi-process.coffee:1:1)
    at /Users/hn/.atom/packages/haskell-ghc-mod/lib/haskell-ghc-mod.coffee:114:16
    at /Users/hn/.atom/packages/haskell-ghc-mod/lib/haskell-ghc-mod.coffee:206:11
    at _this.showTooltip.tooltip (/Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:115:30)
    at /Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:131:7
    at UPIInstance.withEventRange (/Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:208:5)
    at UPIInstance.showTooltip (/Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:130:6)
    at /Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:111:8
    at Function.module.exports.Emitter.simpleDispatch (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:25:14)
    at Emitter.module.exports.Emitter.emit (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:125:28)
    at /Users/hn/.atom/packages/ide-haskell/lib/plugin-manager.coffee:73:18
    at Function.module.exports.Emitter.simpleDispatch (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:25:14)
    at Emitter.module.exports.Emitter.emit (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:125:28)
    at EditorControl.shouldShowTooltip (/Users/hn/.atom/packages/ide-haskell/lib/editor-control.coffee:129:16)
    at /Users/hn/.atom/packages/ide-haskell/lib/editor-control.coffee:53:42

Commands
  5x -1:12.7.0 core:close (atom-text-editor.editor.is-focused)
     -0:11 settings-view:open (atom-text-editor.editor.is-focused)
     -0:07.0 core:close (div.package-detail.panels-item)

Config
{
  "core": {
    "disabledPackages": [
      "git-history",
      "whitespace",
      "build",
      "haskell-pointfree",
      "ide-haskell-repl",
      "ide-haskell-hasktags",
      "ide-haskell-cabal"
    ]
  },
  "haskell-ghc-mod": {
    "additionalPathDirectories": [
      "/Applications/ghc-7.10.2.app/Contents/bin",
      "/Users/hn/.cabal/bin"
    ],
    "enableGhcModi": false,
    "onMouseHoverShow": "Info",
    "onSaveLint": false
  }
}

Installed Packages
# User
autocomplete-haskell, v0.6.2
file-icons, v1.6.13
haskell-ghc-mod, v1.6.0
ide-haskell, v1.3.9
language-haskell, v1.5.2
linter, v1.11.3
minimap, v4.19.0

# Dev
No dev packages

That's weird. Try v1.6.1, it will hopefully at least produce a more meaningful error message.
Note that v1.6.0 tries hard to play nicer with stack. So if you're using stack, you might need to reconfigure haskell-ghc-mod to remove some hacks you were using.

v1.6.1 results in the same error (even the error message is the same).
Stack is configured for this project, but it is not used locally (it is only used on the build server). I have not configured haskell-ghc-mod in any way to work with stack, so I am not aware of any hacks I could remove. Switching the "Stack Sandbox" flag on or off also does not seem to make a difference.

Either I'm misreading stack trace, or something has gone horribly wrong. If v1.6.3 does the same thing (which shouldn't be technically possible), try uninstalling haskell-ghc-mod, restarting Atom, and then reinstalling it again.

Well, there still is an error, but at least now the message is more meaningful:
Atom Version: 1.3.3
System: Mac OS X 10.11.2
Thrown From: haskell-ghc-mod package, v1.6.3
Stack Trace
Uncaught TypeError: Cannot read property 'getInfoInBuffer' of null
At /Users/hn/.atom/packages/haskell-ghc-mod/lib/haskell-ghc-mod.coffee:122

TypeError: Cannot read property 'getInfoInBuffer' of null
    at /Users/hn/.atom/packages/haskell-ghc-mod/lib/haskell-ghc-mod.coffee:122:15
    at /Users/hn/.atom/packages/haskell-ghc-mod/lib/haskell-ghc-mod.coffee:214:11
    at _this.showTooltip.tooltip (/Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:115:30)
    at /Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:131:7
    at UPIInstance.withEventRange (/Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:208:5)
    at UPIInstance.showTooltip (/Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:130:6)
    at /Users/hn/.atom/packages/ide-haskell/lib/upi.coffee:111:8
    at Function.module.exports.Emitter.simpleDispatch (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:25:14)
    at Emitter.module.exports.Emitter.emit (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:125:28)
    at /Users/hn/.atom/packages/ide-haskell/lib/plugin-manager.coffee:73:18
    at Function.module.exports.Emitter.simpleDispatch (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:25:14)
    at Emitter.module.exports.Emitter.emit (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/event-kit/lib/emitter.js:125:28)
    at EditorControl.shouldShowTooltip (/Users/hn/.atom/packages/ide-haskell/lib/editor-control.coffee:129:16)
    at /Users/hn/.atom/packages/ide-haskell/lib/editor-control.coffee:53:42

Commands
     -0:32.7.0 core:copy (atom-text-editor.editor.mini.is-focused)
     -0:32.4.0 core:backspace (atom-text-editor.editor.mini.is-focused)
     -0:28.6.0 core:confirm (atom-text-editor.editor.mini.is-focused)

Config
{
  "core": {
    "disabledPackages": [
      "git-history",
      "whitespace",
      "build",
      "haskell-pointfree",
      "ide-haskell-repl",
      "ide-haskell-hasktags"
    ]
  },
  "haskell-ghc-mod": {
    "additionalPathDirectories": [
      "/Applications/ghc-7.10.2.app/Contents/bin",
      "/Users/hn/.cabal/bin"
    ],
    "enableGhcModi": false,
    "onMouseHoverShow": "Info",
    "onSaveLint": false,
    "debug": true
  }
}

Installed Packages
# User
autocomplete-haskell, v0.6.2
file-icons, v1.6.13
ide-haskell, v1.3.9
ide-haskell-cabal, v1.0.0
language-haskell, v1.5.2
linter, v1.11.3
minimap, v4.19.0

# Dev
No dev packages

Nope. Even more meaningless. This last error should basically be impossible, if package is installed correctly.
Seriously, try reinstalling. Sometimes updates fail for no apparent reason and mess up everything horribly.

The reinstall worked (I uninstalled all packages from atom-haskell, closed Atom, reopened Atom and installed all again).
Thanks for your help!

Glad to hear that.


Visualizations Troubles

I'm unable to get the visualizations to work after a few attempts to debug. I'm able to successfully open up the Jupyter Notebook and run the cells, but my first issue appears when I get to cell 7 or so:
"plt.figure(figsize = (32, 32))
plt.imshow(get_latest_grid('baselines/session_24b49e0b'))"
I've tried changing the session file name and I get the same sorts of errors like so. Has anyone seen this before/know how to get the visualizations up and running? I'm brand new to all of this.
Errors output in Jupyter Notebook when running the two lines in cell 7:
EinopsError                               Traceback (most recent call last)
File ~\anaconda3\Lib\site-packages\einops\einops.py:412, in reduce(tensor, pattern, reduction, **axes_lengths)
411     recipe = _prepare_transformation_recipe(pattern, reduction, axes_lengths=hashable_axes_lengths)
--> 412     return _apply_recipe(recipe, tensor, reduction_type=reduction)
413 except EinopsError as e:
File ~\anaconda3\Lib\site-packages\einops\einops.py:235, in _apply_recipe(recipe, tensor, reduction_type)
233 backend = get_backend(tensor)
234 init_shapes, reduced_axes, axes_reordering, added_axes, final_shapes =
--> 235     _reconstruct_from_shape(recipe, backend.shape(tensor))
236 tensor = backend.reshape(tensor, init_shapes)
File ~\anaconda3\Lib\site-packages\einops\einops.py:200, in _reconstruct_from_shape_uncached(self, shape)
199 if isinstance(length, int) and isinstance(known_product, int) and length % known_product != 0:
--> 200     raise EinopsError("Shape mismatch, can't divide axis of length {} in chunks of {}".format(
201         length, known_product))
203 unknown_axis = unknown_axes[0]
EinopsError: Shape mismatch, can't divide axis of length 6 in chunks of 4
During handling of the above exception, another exception occurred:
EinopsError                               Traceback (most recent call last)
Cell In[8], line 2
1 plt.figure(figsize = (32, 32))
----> 2 plt.imshow(get_latest_grid('baselines/session_24b49e0b'))
Cell In[5], line 3, in get_latest_grid(pth)
1 def get_latest_grid(pth):
2     imgs = np.array([np.array(Image.open(p)) for p in Path(pth).glob('curframe*.jpeg')])
----> 3     grid = rearrange(imgs, '(h2 w2) h w c -> (h2 h) (w2 w) c', w2=4)
4     return grid
File ~\anaconda3\Lib\site-packages\einops\einops.py:483, in rearrange(tensor, pattern, **axes_lengths)
481         raise TypeError("Rearrange can't be applied to an empty list")
482     tensor = get_backend(tensor[0]).stack_on_zeroth_dimension(tensor)
--> 483 return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
File ~\anaconda3\Lib\site-packages\einops\einops.py:420, in reduce(tensor, pattern, reduction, **axes_lengths)
418     message += '\n Input is list. '
419 message += 'Additional info: {}.'.format(axes_lengths)
--> 420 raise EinopsError(message + '\n {}'.format(e))
EinopsError:  Error while processing rearrange-reduction pattern "(h2 w2) h w c -> (h2 h) (w2 w) c".
Input tensor shape: (6, 144, 160, 3). Additional info: {'w2': 4}.
Shape mismatch, can't divide axis of length 6 in chunks of 4

I came here to open my own issue, but I'll just drop this here:
Here are a few things I encountered while trying to do visualization.

there was a line like f tqdm in the imports. This is probably supposed to be from tqdm import tqdm?
as you mentioned, the session name is hardcoded in many places, so you need to replace all of those.
if you have customized num_cpu, look for places where 44 is hardcoded in the visualization and update those
there is also a rearrange call that assumes that num_cpu is divisible by 4. the 4 there needs to be changed to a divisor of num_cpu

After fixing those up, things would be mostly working.

try this and lemme' know if works:
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as mplcm
import matplotlib.colors as colors
from einops import rearrange
from PIL import Image
from pathlib import Path
from sklearn import linear_model
from tqdm import tqdm  # Import tqdm here
def load_run_data(run_path):
run_lists = []
for p in Path(run_path).glob('all_runs_*.json'):
with open(p, 'r') as f:
run_lists.append(json.load(f))
all_runs = [val for tup in zip(*run_lists) for val in tup]
print(f'runs loaded: {len(all_runs)}')
return all_runs
def plot_runs(all_runs, use_keys=None, plot_fit=False, plot_line=False, start=0, group_runs=1, agg_func=np.mean):
all_runs = all_runs[start:]
x = np.arange(len(all_runs) // group_runs)
def add_plot(y, name, alpha, size, plot_line, plot_fit):
    plt.scatter(x, y, alpha=alpha, s=size, label=name)
    if plot_line:
        plt.plot(x, y, alpha=alpha, label=name)
    if plot_fit:
        regr = linear_model.LinearRegression()
        regr.fit(x.reshape(-1, 1), y.reshape(-1, 1))
        lin_y = regr.predict(x.reshape(-1, 1))
        plt.plot(x, lin_y, linewidth=3)

metrics = {m: [run[m] for run in all_runs] for m in all_runs[0].keys()}
metrics['total'] = [sum([v for _, v in run.items()]) for run in all_runs]

if group_runs:
    for m, dat in metrics.items():
        metrics[m] = agg_func(np.array(dat).reshape(-1, group_runs), 1)

available_keys = metrics.keys()
print(f'available metrics: {list(available_keys)}')

if use_keys is None:
    use_keys = available_keys

NUM_COLORS = len(use_keys)
cm = plt.get_cmap('Dark2')
cNorm = colors.Normalize(vmin=0, vmax=NUM_COLORS - 1)
scalarMap = mplcm.ScalarMappable(norm=cNorm, cmap=cm)
fig = plt.figure(figsize=(12, 8))

ax = fig.add_subplot(111)
ax.set_prop_cycle(color=[scalarMap.to_rgba(i) for i in range(NUM_COLORS)]

for i, m in enumerate(use_keys):
    add_plot(np.array(metrics[m]), m, 1.0, 3.0, plot_line, plot_fit)

plt.title('Reward over runs PPO')
plt.legend(bbox_to_anchor=(1.04, 0.5), loc="center left", borderaxespad=0)
plt.show()

def get_latest_grid(pth):
imgs = np.array([np.array(Image.open(p)) for p in Path(pth).glob('curframe*.jpeg')])
grid = rearrange(imgs, '(h2 w2) h w c -> (h2 h) (w2 w) c', w2=4)
return grid
def histo_plot(axis, stat, dat_chunks, version, compare_count):
test_dat = np.array([run[stat] for run in dat_chunks[version]])
n, bins, patches = axis.hist(test_dat, 10, density=False, facecolor="orange", alpha=0.95)
for i in range(compare_count):
    comp_version = max(version - 1 - i, 0)
    compare_dat = np.array([run[stat] for run in dat_chunks[comp_version]])
    n, bins, patches = axis.hist(compare_dat, 10, density=False, facecolor="grey", alpha=0.2 * 0.25**i)

axis.set(xlabel=f"{stat}")

def plot_all(dat_chunks, version, compare_count):
fig, axs = plt.subplots(2, 4, figsize=(16, 9), dpi=120)
fig.suptitle(f"Version {version} Rewards")
for stat, ax in zip(dat_chunks[0][0].keys(), axs.flatten()):
histo_plot(ax, stat, dat_chunks, version, compare_count)
return fig
Debugging your code:
load_run_data('baselines/session_4da05e87_main_full')  # Check if data loads correctly
plot_runs(load_run_data('baselines/session_4da05e87_main_full'), use_keys=None)  # Debug plot_runs
plot_all(chunks(full_dat, 44), 10, 5)  # Debug plot_all

rearrange

This helped a ton!
I'm just playing around with things and only have a single 6 core CPU, so this solved a ton of potential problems all at once! I can see some pretty pictures now!

Hi yes the code arranges the screens into a grid and you'll get that error if your number of screens/games isn't divisible by the grid dimension. Glad you were able to get it working :)
Code could probably use an automatic check though.

Glad that it works

It is not unusual for the performance to decrease or stop working, the optimization can be quite fragile. You're using much less environments than the original version, so you can try tweaking parameters to get it learning better. You could try lowing the discount factor (gamma) a little, or changing episode length, or try to start by disabling all rewards except for exploration. Might take some experimentation but eventually you should see the reward start to climb!
Also documentation for PPO in stable baselines:
https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html


[PE-D][Tester C] Label for DG incorrect for view sequence diagram

Labels: type.DocumentationBug severity.Medium
original: GuanpengR/ped#9

Fixed. Thank you!


Quorum Reader - Include Primary to Meet Quorum when One Secondary Replica is Non-Responsive

Problem:
With our Bounded Staleness consistency settings, where it normally reads data from two secondary replicas and chooses the most recent version. It then verifies consistency by sending "head requests" to all secondaries (and the primary if the replica set is less than 4).
Recently, during a deployment, one secondary became unavailable due to the update, and another crashed. This left us with a replica set of only three (two secondaries and the primary).
The system attempted to read from the two remaining secondaries, but one was unreachable due to the crash. This triggered a validation check, which normally would involve reading from the primary if no data was retrieved from the secondaries. However, in this case, the validation logic prevented reading from the primary because the replica set size was 3 (it expected at least 2 responses for a quorum).
This validation failure caused an exception and retries, but it didn't resolve the issue.
Proposed Solution:
To avoid this issue, we propose modifying the system's behavior when the replica set size is reduced and one secondary is unavailable. Instead of requiring a quorum from the remaining secondaries, we would include the primary in the selection process. This would allow the system to read from all available replicas and establish consistency. This change would ensure the system remains operational even during similar failures.

When fixing this please also make the error messages of the GoneExceptions more useful (currently PartitionId is missing for example)
https://msdata.visualstudio.com/CosmosDB/_git/CosmosDB?path=/Product/Microsoft.Azure.Documents/SharedFiles/QuorumReader.cs&version=GBmaster&line=139&lineEnd=144&lineStartColumn=1&lineEndColumn=1&lineStyle=plain&_a=contents


SDK Build Tools revision is too low

ReactNative 0.56 changed the SDK level for android builds to 25.
Expected Behavior
Including the library as described in the Readme works for a newly created project on<EMAIL_ADDRESS>and Gradle (or react-native run-android for that matter) is able to build the app.
Current Behavior
The build fails with the following error:
The SDK Build Tools revision (23.0.1) is too low for project ':react-native-sqlite-storage'. Minimum required is 25.0.0
Possible Solution
#279 mentioned a solution which does nothing but changing the setting on react-native-sqlite-storage inside node_modules and will be overwritten on a new npm/yarn install (not to mention a CI scenario or other collaborators).
A solution (I'll send a PR in a second) could be to fetch the SDK versions from the root project:
def safeExtGet(prop, fallback) {
    rootProject.ext.has(prop) ? rootProject.ext.get(prop) : fallback
}

android {
    compileSdkVersion safeExtGet('compileSdkVersion', 26)
    buildToolsVersion safeExtGet('buildToolsVersion', '26.0.3')

    defaultConfig {
        minSdkVersion safeExtGet('minSdkVersion', 16)
        targetSdkVersion safeExtGet('targetSdkVersion', 26)
        versionCode 1
        versionName "1.0"
    }
    lintOptions {
        abortOnError false
    }
}

Steps to Reproduce (for bugs)

react-native init with version 0.56
yarn add react-native-sqlite-storage + Follow readme to add it to the Android project
cd android
./gradlew assembleDebug -> Fails

Context
This blocks me from adding typeorm to my current react native project. Will block others who use the most current react-native version or who manually updated their sdk versions, too.
Also:

Google Play will require that new apps target at least Android 8.0 (API level 26) from August 1, 2018, and that app updates target Android 8.0 from November 1, 2018. - Source

Your Environment

React Native SQLite Storage Version used: 3.3.6
React Native version used: 0.56.0

Your PR is merged.


[C++][Parquet] Make parquet-format a git submodule and add tool for updating generated Thrift code

As a follow up to PARQUET-449
Reporter: Wes McKinney / @wesm
Note: This issue was originally created as PARQUET-466. Please see the migration documentation for further details.

Wes McKinney / @wesm:
In the interest of parsimonious development, I propose to settle (for now) for periodically syncing parquet.thrift with upstream parquet-format and automatically generate the Thrift bindings as part of the build process (as described in PARQUET-468) rather than checking in the generated bindings (which may indeed by Thrift version-dependent as we've seen).

Wes McKinney / @wesm:
We can keep parquet.thrift updated manually until it becomes an issue.

Wes McKinney / @wesm:
This can be resolved when https://github.com/apache/parquet-cpp/pull/55 is merged


communication

The communication plan needs to be updated to reflect the latest changes in the team schedule and channels. This involves:

Adding new communication channels
Updating the schedule for meetings
Clarifying the roles of team members

seems good ! nice work

i'm all for it!! GREAT WORK!


Implementation docs?

There's a design page in the Sphinx docs, but so far it's pretty thin: https://colcon.readthedocs.io/en/released/developer/design.html
I'm just getting started as a colcon user, but also interested in questions around what its capabilities and limitations are as a framework, once I am at a comfort level with it to begin hacking on extensions, etc. For example:

What is the story with setup, local_setup, and _local_setup_util? Which of these suites of files are being generated by colcon core vs the build type plugin vs the package's build system? Why is the build ordering logic located in _local_setup_util_sh.py, a file generated into the installspace?
How is the environment being prepared for a particular package's build? In the non-merged (default) install case, how are the install trees of the build dependencies ending up merged together? Does the merging work properly even if I --start-with a package partway through the build? What if I don't have the source for that package and only a cached/saved installspace that has local setup files? (these last two cases do not work properly with catkin_tools)
It looks like a bunch of the environment setup (prepending variables and so-on) is being handled by the files in share/[pkg]/hook/*.dsv. This isn't a catkin concept though; is it a colcon thing? It appears these are mostly generated from templates originating in colcon-core, but these seem pretty specific to the buildsystem in question. What's going on here?
What is happening during the big pause at the beginning of a colcon build invocation? I'm assuming it's scanning the workspace to discover packages, but I'd love to know more about where the logic for that lives and how it works (and why it takes 10+ seconds for ~600 packages). Part of this is that it would be great to be able to run that once and cache it for successive invocations, but I'm also curious about whether the discovery stuff could be extracted into a form usable by rosdistro, such that packages listed as source in a distribution.yaml wouldn't necessarily need a package.xml to be patched into them?

It would be great to have some high-level explanation of what the steps are that colcon is taking to execute a build, what is managing what in terms of actions and responsibilities.

There's a design page in the Sphinx docs, but so far it's pretty thin: https://colcon.readthedocs.io/en/released/developer/design.html

Agreed, it certainly could benefit being extended. I will answer some questions inline below. For some I will reference PRs I created for the docs based on this ticket. If you have further questions like this please keep them coming. I will try to either answer them directly or convert the information into docs.

What is the story with setup, local_setup, and _local_setup_util? Which of these suites of files are being generated by colcon core vs the build type plugin vs the package's build system?

All *setup* files in the install prefix path are coming from colcon and colcon alone. Whenever a build system tries to write similar setup files into the install prefix (e.g. catkin) the generation is being suppressed (e.g. for catkin packages by passing -DCATKIN_INSTALL_INTO_PREFIX_ROOT=0).

Why is the build ordering logic located in _local_setup_util_sh.py, a file generated into the installspace?

The logic embedded in that Python file to order packages topologically is necessary when setting up the environment of packages too. The environment setup of a package might depend on the environment of its dependencies already been setup. Generating a statically ordered list of packages doesn't work when installing packages in multiple iterations into the same install space. Therefore the script first discovers the packages in the install space, orders them topologically and then sources the package specific setup files in that order.

How is the environment being prepared for a particular package's build? In the non-merged (default) install case, how are the install trees of the build dependencies ending up merged together?

When a package is being built it gets the environment defined by all its recursive dependencies. For each of these packages the environment is setup in topological order. This only needs to consider the packages in the workspace since other packages (external or from an underlay) require to have been sourced beforehand.
As a consequence if a package doesn't declare a build dependency it should fail to find its resources (assuming a not merged install space).

Does the merging work properly even if I --start-with a package partway through the build?

Yes. That being said must have built dependency packages at least once before you can build packages on top. But that could have happened in a previous invocation.
The install space doesn't store any information specific to the set of processed packages. It just has the artifacts of all previously installed packages and sources them in topological order.

What if I don't have the source for that package and only a cached/saved installspace that has local setup files? (these last two cases do not work properly with catkin_tools)

I am not sure I understand the question. I will try to but maybe I miss the point. Please rephrase then so I can answer it well.
While workspaces can be chained each workspace is being sourced in the order of the chaining. So first all packages within an underlay, then all packages within an overlay.
If you install two different workspace into the same install space then sourcing the install space considers all packages in that install spaces and sources them in topological order (it doesn't matter how that install space was populated).

It looks like a bunch of the environment setup (prepending variables and so-on) is being handled by the files in share/[pkg]/hook/*.dsv. This isn't a catkin concept though; is it a colcon thing? It appears these are mostly generated from templates originating in colcon-core, but these seem pretty specific to the buildsystem in question. What's going on here?

Please see colcon/colcon.readthedocs.org#51 which describes the general environment setup as well as the role of .dsv files.

What is happening during the big pause at the beginning of a colcon build invocation? I'm assuming it's scanning the workspace to discover packages,

Correct. Recently a few changes have landed (but not yet released) which improve the performance of the involved code quite a bit (#256, #257, #259) but the time spent in this phase is still significant for larger workspaces.

but I'd love to know more about where the logic for that lives and how it works (and why it takes 10+ seconds for ~600 packages).

Similar to the previously referenced PRs there is probably potential to reduce the overhead by improving data structures and overall flow. It would be best to profile such a use case (as done for the past PRs) to identify the most significant bottlenecks - maybe persisting the parsed XML results could be a fruitful option (just speculating).
It would certainly help if that part would be parallelized (as the actual build is).

Part of this is that it would be great to be able to run that once and cache it for successive invocations,

That sounds not trivial given the number of things which could change and would affect the result (most of all any change in the file system in locations being considered by the discovery step). If we would want to go that route we should make sure that this can be realized through an extension (which the core probably will need to add a new extension point for?).

but I'm also curious about whether the discovery stuff could be extracted into a form usable by rosdistro, such that packages listed as source in a distribution.yaml wouldn't necessarily need a package.xml to be patched into them?

One of the slides from the recent ROSCon presentation briefly mentions the parts involved in this (see https://github.com/dirk-thomas/slides_roscon2019_colcon):

Discovery: determine directories to check for a package, commonly crawling recursively.
Identification: determine if a directory contains a package as well as its name and type (the latter deciding the recipe how it should be processed)
Augmentation: add metadata to identified packages (e.g. additional dependencies which couldn't be extracted from the identification logic since they are not available in a machine readable form)

Each of these steps is encapsulated by an extension point in colcon which is ultimately being implemented by multiple extensions across different packages. E.g. colcon-ros provides an identification extension based on package.xml files). The general design goal in colcon is that all of this logic should be accessible through API. So it should be feasible to reuse it from other Python code. (Not sure if it would be desired to do that from rosdistro and what the implications would be - e.g. the information only being available after cloning the package in full and running the Python logic.)

It would be great to have some high-level explanation of what the steps are that colcon is taking to execute a build, what is managing what in terms of actions and responsibilities.

Please see colcon/colcon.readthedocs.org#52 for the high level program flow of the list and build verb. Additionally colcon/colcon.readthedocs.org#53 briefly enumerates the most important extension points. I hope that helps to get a better overview how things fit together.

Another aspect of colcon about which I was unable to learn anything from the existing docs is the Observer system. I'm working on a plugin which implements a new verb that has nothing to do with building and doesn't use the build space argument. It mostly works, but I get this angry message upon completion of my verb's executor run:
[0.608s] ERROR:colcon.colcon_core.event_reactor:Exception in event handler extension 'compile_commands': 'Namespace' object has no attribute 'build_base'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/event_reactor.py", line 78, in _notify_observers
    retval = observer(event)
  File "/usr/lib/python3/dist-packages/colcon_cmake/event_handler/compile_commands.py", line 47, in __call__
    package_level_json_paths = self._get_package_level_json_paths()
  File "/usr/lib/python3/dist-packages/colcon_cmake/event_handler/compile_commands.py", line 97, in _get_package_level_json_paths
    json_path = self._get_path(package_name)
  File "/usr/lib/python3/dist-packages/colcon_cmake/event_handler/compile_commands.py", line 114, in _get_path
    path = Path(self.context.args.build_base)
AttributeError: 'Namespace' object has no attribute 'build_base'

I can silence this by patching in context.args.build_base = '', but I'd like to understand better where this observer thing came from and why it is here. Being in the colcon_cmake package, it appears connected to the build_type, but I can't see what it is I'm invoking that would have prompted this thing to exist or be observing, or even the event to have occurred.
It appears that https://github.com/colcon/colcon-cmake/issues/81 raises a similar issue, but my immediate concern is more just about understanding the relationship between these event handlers and the rest of the system.

Another aspect of colcon about which I was unable to learn anything from the existing docs is the Observer system. I'm working on a plugin which implements a new verb that has nothing to do with building and doesn't use the build space argument.

Anything happening during the invocation of colcon is represented as an event and pushed into a queue. E.g. a package starts/ends, a subprocess is invoked / returned, a subprocess generates stdout / stderr output, etc. Some event types can be found in colcon-core.
The event reactor (implementing the reactor pattern) processes each event from that queue and passes it to various event handlers (which implement the EventHandlerExtensionPoint). Those contain the actual logic to e.g. print the information of an event on the console or to a log file.

It mostly works, but I get this angry message upon completion of my verb's executor run:

The CompileCommandsEventHandler is the one raising an exception. It is used to build a compile_commands.json file after an invocation. Conceptionally that only needs to happen after a build but unfortunately the event handler doesn't have the necessary context to not perform the action in other cases - either for test as described in colcon/colcon-cmake#81 (which is just unnecessary resource usage) or in your case for a custom verb (where it fails since there is no build_base argument).
For now there wasn't a strong reason to resolve the issue since all other verbs had a build_base so beside the extra resource usage this wasn't a problem. Your temporary workaround is certainly one way to about the error message. Another temporary solution would be to disable the extension in question by setting the environment variable COLCON_EXTENSION_BLOCKLIST=colcon_core.event_handler.compile_commands.
Anyway I just created colcon/colcon-cmake#88 which should resolve the original issue and I would hope also addresses the problem with your custom verb. It would be great if you could give it a try and comment if that patch works for your use case.

That is helpful to understand, and thanks for putting up the fix so promptly.
Relatedly, I have a situation where I would like to be able to plug in some extra per-package pre/post build functionality to colcon (think analytics, reporting, etc, but could also be heavier-weight stuff like packaging-related tasks). I don't want to have to individually wrap each build task to insert this extra functionality, nor do I want to have to hack on the build verb.
Is listening for JobStarted/JobEnded events and just inserting the work there a reasonable thing to do? Or is there a better way to handle this that exists (or is planned)? All the event handlers discussed here seem to be super lightweight and related to logging/reporting.

Is listening for JobStarted/JobEnded events and just inserting the work there a reasonable thing to do?

It probably depends on what exactly you want to do. For light stuff which can happen asynchronously to the ongoing build that sounds fine to me.
For heavier-weight stuff the issue is that all events are being processed sequentially and long running blocking event handler would negatively affect / delay all following events. Maybe you could overload the work into a thread started by the event handler and make sure to join the thread when the EventReactorShutdown is handled.
One big downside of the whole approach is that there is no clear way to interact with the user since you can't easily generate / post events from that logic. So it feels more like the wrong way of doing it.

Or is there a better way to handle this that exists (or is planned)?

Atm there is no plan to modify / extend anything in that area. But with enough detailed use cases we could come up with a plan how they could be handled best and what would need to be changed / added to enable them. Maybe you can try to describe the desired tasks with some details to get a better idea about the desired features?

All the event handlers discussed here seem to be super lightweight and related to logging/reporting.

graphviz_anim is probably the most extreme one since it might take several seconds (if not longer) but since it is happening at the very end of the invocation its isn't too harmful.

Perhaps there is an opportunity for a TaskAugmentationExtensionPoint which permits arbitrary code to be run pre-Task and post-Task, in a way that is properly reflected in the status bar, generates log files, etc?

As a kind of cross cutting aspect - that sounds like an interesting thought. You would probably have to balance what kind of work should be done in those since they would need to be applicable to any kind of task (build vs. test, cmake / python / ament / ros pkgs).

You would probably have to balance what kind of work should be done in those since they would need to be applicable to any kind of task (build vs. test, cmake / python / ament / ros pkgs).

It would be important for the augmenting function/plugin to have some insight into what the verb/builder/task is— could be as simple as just passing that information through and letting it short return if its functionality isn't applicable. Or there could be a mechanism to pre-filter, could be as simple as just a regex against the name or something. Certainly to be useful, it would need to have access to whatever the TaskContext is from the associated task.
I still don't have as clear a sense how all this works in Colcon-land. The catkin_tools approach is very declarative in this regard, with explicit, named CommandStage/FunctionStage instances that have to defined upfront, whereas it seems that fully embracing the coroutine model has meant colcon can be more flexible with basically just running arbitrary Python code to drive the build, and yielding whenever a subprocess or whatever else happens.
As far as my use-case, I'm experimenting with tarring up the installspaces after build and sending them off to a cache where they can be pulled in to pre-cook the next build. The overall operations are:

Spider the installspace, hardlink all files into a parallel structure elsewhere (so that I can get some naming/hierarchy that I want).
Prepare an archive of this structure.
Upload the archive.

Currently I'm doing this in a separate verb, invoked when the build is complete, however, there are several reasons to want to do it in an interleaved manner—

Assets will be available for consumption by other builds sooner if they're uploaded sooner.
The upload will be gated on network IO, since the main build doesn't use the network at all, it would be nice to spread this throughout.

So yeah, I'm not sure what other use cases might justify the addition of a formal extension point here, but if you think it might have legs, I can open a separate issue to discuss further. In the meantime, I'll maybe wander down the path of kicking this work onto a separate background thread that manages its own logging and so-on.

Hi,
Additional doc would be welcome on using --symlink-install,
I created a pure cmake (no ament nor catkin) package and I am installing some python scripts from the source folder.
I would like these to be installed as symlinks when using colcon build --symlink-install.
Though in the documentation the information is very limited.
Thanks for your help.

@MaximilienNaveau --symlink-install requires support from the build system. For a pure CMake package that isn't the case and therefore the installation will copy files. In order to support this feature for your pure CMake package colcon-cmake would first need to pass that flag somehow (likely through an environment variable) and then your CMake code would need to override the install() behavior. That is one of the features ament_cmake provides.


webpackCompilationPlugins option of InjectManifest seems to be ignored

Library Affected:
workbox-build
Browser & Platform:
NA
Issue or Feature Request Description:
webpackCompilationPlugins Option of InjectManifest seems to be totally ignored in my setup.
When looking at the codebase of workbox, the name of this option is only mentionned in 4 places: twice in docs, once in input validation, and once in a test file, nowhere is it used in actual code.
So I don’t understand how this option is taken into account, and it does not seem to be taken into account in my setup.
Any input on this ?

Hello @crubier—the option is used in https://github.com/GoogleChrome/workbox/blob/f6fb94a9d8b922fdcb758129d94e6d2a959f2167/packages/workbox-webpack-plugin/src/inject-manifest.js#L205-L209
Working with webpack's child compiler is notoriously difficult, but my understanding is that the code above should successfully apply any plugins you provide to the compilation that is applied to your swSrc file.
If you have an example of a setup in which a plugin doesn't behave as expected, could you share that configuration?

Hi @jeffposnick ,
Ok well, sorry, I think I found a bug in Github code search feature then aha.
My Github code search gave these result, but indeed the code you are talking about exists (and should have been visible in the search result I expect)

On another topic, after moving my setup to Webpack 5, the problem is solved on workbox side I think (The plugin gets applied, as expected). But the plugins seems to have no effect on the compilation / some hooks of the plugin seems to not be called. I'll keep investigating...

If you have any follow-ups, feel free to post details. I'm going to close this for now, though.


(android) Make the phone number searching smarter

Platforms affected
Android
Motivation and Context
The phone number searching in Android is very poor by default. The phone number needs to be an exact match (or contain only a portion) of the number.
This doesn't work with something like caller ID. For example, the Android caller ID will report the number with a "plus" (+) followed by the country code and then the number with no spaces.
As an example, if I get a phone call from Jenny, her number will show up like "+15058675309". But in my contacts list, I have her number as "(505) 867-5309".
The default search method won't return Jenny's contact information in this case.
Description
This change uses the method described on this Stack Overflow post. This will allow the search to find Jenny's contact information.
It only changes the search if the user is searching by phone number (the fields sent to search have "phoneNumbers" in it), otherwise the search stays the same.
Testing
Tested by searching with and without the phone number.
Checklist

[x ] I've run the tests to see all new and existing tests pass
[ ] I added automated test coverage as appropriate for this change
[x ] Commit is prefixed with (platform) if this change only applies to one platform (e.g. (android))
[ ] If this Pull Request resolves an issue, I linked to the issue in the text above (and used the correct keyword to close issues using keywords)
[ ] I've updated the documentation if necessary

We are archiving this repository following Apache Cordova's Deprecation Policy. We will not continue to work on this repository. Therefore all issues and pull requests are being closed. Thanks for your contribution.


Join failures and bursts of packets

I've been having issues joining today 23rd July.
At midnight 12:00am bst I noticed a burst of data packets coming through. As well as my old joins. This is using staging console.
Device examples
Dev eui
A840417D11829434
1D4A7D0000A49A12

Closing this as we believe will be addressed with later miner release here: https://engineering.helium.com/2021/08/06/blockchain-release-snapshot-update.html


Uncaught TypeError: Cannot read property 'length' of undefined

[Enter steps to reproduce:]

...
...

Atom: 1.38.2 x64
Electron: 2.0.18
OS: Mac OS X 10.14.5
Thrown From: Atom Core
Stack Trace
Uncaught TypeError: Cannot read property 'length' of undefined
At /Applications/Atom.app/Contents/Resources/app/static/<embedded>:14

TypeError: Cannot read property 'length' of undefined
    at ScreenLineBuilder.emitText (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:14:1190071)
    at ScreenLineBuilder.buildScreenLines (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:14:1185777)
    at DisplayLayer.getScreenLines (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:14:89970)
    at TextEditorComponent.queryScreenLinesToRender (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:11:79712)
    at TextEditorComponent.updateSyncBeforeMeasuringContent (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:11:70365)
    at TextEditorComponent.updateSync (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:11:67993)
    at TextEditorComponent.didMouseWheel (/Applications/Atom.app/Contents/Resources/app/static/<embedded>:11:92801)

Commands
 12x -2:43.7.0 core:move-up (input.hidden-input)
 10x -2:40.2.0 core:move-left (input.hidden-input)
  2x -2:38.4.0 core:backspace (input.hidden-input)
  4x -2:36.8.0 core:move-right (input.hidden-input)
     -2:35.6.0 core:backspace (input.hidden-input)
  4x -2:35.2.0 core:move-right (input.hidden-input)
 18x -2:34.4.0 core:select-right (input.hidden-input)
  3x -2:31.7.0 core:backspace (input.hidden-input)
     -2:16.1.0 core:move-right (input.hidden-input)
     -2:15.2.0 core:move-left (input.hidden-input)
     -2:07.7.0 core:backspace (input.hidden-input)
 10x -2:06.7.0 core:move-left (input.hidden-input)
 10x -2:02.9.0 core:move-right (input.hidden-input)
     -2:00.2.0 core:move-left (input.hidden-input)
  3x -1:58.4.0 core:save (input.hidden-input)
     -1:07.5.0 core:backspace (input.hidden-input)

Non-Core Packages
atom-file-icons 0.6.0
atom-html-preview 0.2.6
autocomplete 0.47.0
export-html 3.1.0
japanese-menu 1.14.0
language-fortran 2.1.7
minimap 4.29.9
platformio-ide-terminal 2.9.1
project-manager 3.3.7
sync-settings 0.8.6

Thanks for contributing!
We noticed that this looks like a duplicate of https://github.com/atom/atom/issues/19108 so you can subscribe there for updates if you'd like.
Based on previous reports, it seems like the error isn't reproducible if you close and reopen the file or project that's giving you trouble.  But if you have specific repro steps with a minimal example or any additional details, please let us know in https://github.com/atom/atom/issues/19108.

Because we treat our issues list as the Atom team's backlog, we close duplicates to focus our work and not have to touch the same chunk of code for the same reason multiple times. This is also why we may mark something as duplicate that isn't an exact duplicate but is closely related.
For information on how to use GitHub's search feature to find out if something is a duplicate before filing, see the How Can I Contribute? section of the Atom CONTRIBUTING guide.


api: clarify Command/Args distinction

To address some issues with using Command/Args, the field
documentation has been updated to clarify that Command should only be
used when overriding Entrypoint. For most docker uses, only Args
should ever be used, as is done in runc.
Args is now appended to Command if both are present.
A --command flag has been added to be able to test that the feature is
working correctly across systems.
Signed-off-by: Stephen J Day<EMAIL_ADDRESS>cc @aluzzardi

Raft test failures look unrelated. I will take a look.

[Current coverage][cc-pull] is 54.05%

Merging [#1092][cc-pull] into [master][cc-base-branch] will decrease coverage by <.01%

@@             master      #1092   diff @@
==========================================
  Files            75         75
  Lines         11717      11717
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           6335       6334     -1
- Misses         4472       4473     +1
  Partials        910        910

[][cc-pull]

Powered by Codecov. Last updated by [114f9b5...494b528][cc-compare]
[cc-base-branch]: https://codecov.io/gh/docker/swarmkit/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/docker/swarmkit/compare/114f9b5f972c96b5686c56d6b9c3cc2a4d1a293c...494b5289b4f55fda0c34fa700584b0a42fbf3688
[cc-pull]: https://codecov.io/gh/docker/swarmkit/pull/1092?src=pr

LGTM

@stevvooe Can you coordinate with @sfsmithcha and make sure this makes it somehow into the docs?


Pipeline mode

Implement client-side pipelining of requests.
When executing a text query or a prepared statement, the following packet exchange occurs:
     client                                server
             ----- query request --->
             <-------   query OK ----
             <-------   metadata ----
             <-------   rows --------
             <-------   EOF ---------

Currently, connection::query or prepared_statement::execute write the query request, read the OK and the metadata. Then the client needs to read all the rows until the EOF before executing another query request.
It is possible to write several of these query requests at once, in batch. Doing this, we can save round-trips. Subsequent queries won't wait until a response has been written. Note that there is no special support for this neither in the protocol nor in the server. The server will process the queries sequentially, as if they had been written one by one. If one of the queries fail, subsequent queries will be executed, regardless of the result of previous queries.
This would be the message exchange:
     client                                server
             --- query request 1 --->
             --- query request 2 --->
             <-----   query 1 OK ----
             <-----   metadata 1 ----
             <-----   rows 1 --------
             <-----   EOF 1 ---------
             <-----   query 2 OK ----
             <-----   metadata 2 ----
             <-----   rows 2 --------
             <-----   EOF 2 ---------

The client would still need to read the resultset yielded by query 1 completely before starting with query 2 resultset.
Implementing this pipeline mode requires an additional, lower level interface. A possible implementation for requests is a serializer object with methods such as add_query and add_statement_execution that implements request serialization, then a connection::write(serializer) method. To read responses, we can implement methods such as connection::read_resultset_metadata(resultset&) and connection::read_row(resultset&, row&).

MariaDB has a bulk execution interface that may be relevant here: https://mariadb.com/kb/en/com_stmt_bulk_execute/


Re-sizeable progress indicator

Add support for re-sizeable progress indicators, both for smaller to be used inside components and larger for "page loading state".

[x] Verify which progress indicator we want to be re-sizeable.

Requested are inside components such as checkboxes (selection controls?), textfields(helper or meta text?), buttons
Requested re-sizeable progress indicators are circular and dots ( Can others work? )~

[x] Verify with design what we want to change, support and valid sizes? Same size-scale as Icon?
[ ] Implement the stuff. Make sure you conform with other components on api.

Loading states inside components may just need to be an example of components being swapped.
Avoid bloated "loading" props if possible

[ ] Reward yourself with cake on a job well done 🍰

Notes from meeting with @lucasveil

We will introduce size property to circular and star progress indicators
Size will follow same scaling as defined for Icon

resolves #417
resolves #1041

Providing a re-sizeable progress indicator and testing usage in button uncoverd missing color variants, resolved in #1092

Providing a re-sizeable progress indicator and testing usage in button uncoverd missing color variants, resolved in #1092


Setting Up yargs.js.org

Here is the hit list to get this done.

[x] You create a gh-pages branch I can submit a pull request to
[ ] I get that Flatdoc site set up
[x] You submit a PR to https://github.com/js-org/dns.js.org to add yargs.js.org to cnames_active.js
[ ] Rejoice!

awesome, I will get on this tomorrow.

@blainsmith alright, gh-pages branch is a thing.

This looks fun. Let me see if I can prep the gh-pages branch a bit.

I recreated the gh-pages branch and setup the basic Flatdoc template.
http://bcoe.github.io/yargs/
Would be cool to customize the default theme. Not sure what you had in mind @blainsmith, but this should get us going. Thanks!

\o/

LOL!  Glad it was such a piece of :cake:  to do.

@blainsmith Your todo list made it easy. Thanks a lot!
If anyone has ideas for how to customize the Flatdoc site, I'm all ears (and PRs are welcomed)!

Should be as simple as adding your own stylesheet with overrides.

@blainsmith Thanks again!

Anytime!


SDK - SvelteKit integration

SvelteKit is built on Svelte, a UI framework that uses a compiler to let you write breathtakingly concise components that do minimal work in the browser, using languages you already know — HTML, CSS and JavaScript
fast:
Powered by Svelte and Vite, speed is baked into every crevice: fast setup, fast dev, fast builds, fast page loads, fast navigation. fastest choice for an TMA.
fun:
No more wasted days figuring out bundler configuration, routing, SSR, CSP, TypeScript, deployment settings and all the other boring stuff.
flexible:
SPA? MPA? SSR? SSG? Check. SvelteKit gives you the tools to succeed whatever it is you're building. And it runs wherever JavaScript does.

Svelte seems good to me, but the problem is as long as I am the only one regular contributor, I need some time to dive a bit in Svelte. This would also be cool if we could find other developers with experience in Svelte to evaluate the implementation.
We will keep this issue opened not to lose it and return after some time

I'd be interested to help or contribute with a Svelte integration. I think that the target version should be Svelte 5 (unstable at the moment), that will be backward compatible with Svelte 4, allowing to use the library in both versions.

I just have started development of TMA with sveltekit. We'll see how it goes. I am also interested in contributing and maintaining sveltekit integration

Svelte seems good to me, but the problem is as long as I am the only one regular contributor, I need some time to dive a bit in Svelte

@heyqbnk you could reach me and we can work on this together :)

Any ideas about routing?

Yeah, I'm also down to contribute to the SvelteKit SDK.

@ZerdoX-x I have used SvelteKit native routing without issues. On firsts mini-apps releases (when they were called web apps) there were issues navigating, and I had to use goto(...), but that is not longer the case.

how is it going with the svelteKit tma SDK? I'm down to contribute as well!

Hey! I would like to ask you to pausethe process a bit as long as I am currently working on a new major version. It will be completed approximately in 1-2 weeks

@heyqbnk let me know if there is any help needed

Hey! I forgot to remember, that SDKv2 is ready.
You can implement the SvelteKit integration using the SDK for React as a reference

@telegram-apps/sdk-svelte has just released. Thank you for your time
https://docs.telegram-mini-apps.com/packages/telegram-apps-sdk-svelte


Optionally suppress support library classes

When using AndroidX the old support library classes are essentially fake news. Could we suppress them with an optional toggle? To do this reliably the app might download and parse androidx-class-mapping.csv from https://developer.android.com/topic/libraries/support-library/refactor.
Alternatively the search results could be presented in a fancier consolidated format where the three classes are collapsed together with separate affordances for platform, support, and AndroidX. It may not be feasible to collapse platform classes without a canonical mapping between those and support / AndroidX.

I'm not a big fan of options. Let me ask around and see how long we plan to return them in the search results. I'd rather see the docs updated so that the androidx versions point to the dejetified versions but they don't show up in search results.

I echo @jameswald 's sentiment. It'd be great if it can be tweaked so that the "preferred" results are more accessible. E.g., assuming androidx is the preferred one, I hope the app would

reorder so that androidx classes show up on top, followed up platform, then support, or
filter out so that support (or even platform).

A screenshot (of current behavior), to help to visualize desirable result:

Yeah let's fix this in sort for now.
On Thu, Dec 27, 2018 at 12:35 AM Sam Lee<EMAIL_ADDRESS>wrote:

I echo @jameswald https://github.com/jameswald 's sentiment. It'd be
great if it can be tweaked so that the "preferred" results are more
accessible. E.g., assuming androidx is the preferred one, I hope the app
would

reorder so that androidx classes show up on top, followed up
platform, then support, or
filter out so that support (or even platform).

A screenshot (of current behavior), to help to visualize desirable result:
[image: image]
https://user-images.githubusercontent.com/250644/50466936-2179a080-0955-11e9-8093-28af201e6c57.png
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/JakeWharton/SdkSearch/issues/111#issuecomment-450078439,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEEEZbS4_n3xefrfU8gj-hwuYbjOvRHks5u9FwZgaJpZM4VPoiU
.

I have removed them from the next version. Let's see if anyone complains 😀 .

As one vote, I work on some projects use androidx, while some others still use android support library: so I have some reservation on outright removal of support library.

Is there a meaningful difference to visiting the documentation for the support library type instead of the AndroidX type? Both types are created from the same source file so there should be no functional difference nor difference in documentation.

@JakeWharton I always assume that while support and Androidx start out the same, at some point I'll start to see methods / classes that are in Androidx only.
From what you suggest, they are still the same (and will stay the same for sometime), then there is no reason to visit support library doc.

Well that's a decent point. Now with the betas there are new APIs being
added.
On Wed, Feb 27, 2019, 6:14 PM Sam Lee<EMAIL_ADDRESS>wrote:

@JakeWharton https://github.com/JakeWharton I always assume that while
support and Androidx start out the same, at some point I'll start to see
methods / classes that are in Androidx only.
From what you suggest, they are still the same (and will stay the same for
sometime), then there is no reason to visit support library doc.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JakeWharton/SdkSearch/issues/111#issuecomment-468069118,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEEEdgsHESo_BN5miC_IFLaOP4mZzkPks5vRxFGgaJpZM4VPoiU
.

I just came across an AndroidX-only class while working on a project using the old support library:
https://developer.android.com/reference/androidx/test/core/app/ApplicationProvider


[Bug]: workspace vs workspace_base issue?

Is there an existing issue for the same bug?

[X] I have checked the troubleshooting document at https://opendevin.github.io/OpenDevin/modules/usage/troubleshooting
[X] I have checked the existing issues.

Describe the bug
When the codeact agent tries to write jupyter code and runs it, I get the error: bash: line 1: /workspace/.tmp/.ipython_execution_tmp.py: No such file or directory
When I look at the "code editor" tab, the .tmp/.ipython_execution_tmp.py is there but under workspace_base

My guess is that's the issue?
Current Version
opendevin:0.5

Installation and Configuration
Doesn't seem to matter how I run it but this is my current starting command:
export LLM_API_KEY="..."
export LLM_MODEL="..."
export LLM_BASE_URL="..."
export WORKSPACE_BASE=$(pwd)/open_devin_test

sudo docker run \
    -e LLM_API_KEY \
    -e LLM_MODEL \
    -e LLM_BASE_URL \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e SANDBOX_TYPE=exec \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    ghcr.io/opendevin/opendevin:0.5

I'm running this within wsl2 on windows

Model and Agent
Behavior doesn't vary by model, have only ever tried it with the code act agent
Reproduction Steps

Launch open devin
Prompt it to write python code
Get error

Logs, Errors, Screenshots, and Additional Context

@williambarberjr what do you get if you run ls -lah $(pwd)/open_devin_test

any solution?

export WORKSPACE_BASE=$(pwd)/workspace

but no files in the container.

seems the sandbox is not restarting?

i have always these problem
I'm running this on Mac m1
`export WORKSPACE_BASE=$(pwd)/workspace;
docker run
--pull=always
-e SANDBOX_USER_ID=$(id -u)
-e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE
-v $WORKSPACE_BASE:/opt/workspace_base
-v /var/run/docker.sock:/var/run/docker.sock
-p 3000:3000
--add-host host.docker.internal:host-gateway
ghcr.io/opendevin/opendevin:0.5`

@williambarberjr what do you get if you run ls -lah $(pwd)/open_devin_test

ls -lah $(pwd)/open_devin_test
total 12K
drwxr-xr-x  3 myusername myusername 4.0K May 10 13:58 .
drwxr-x--- 27 myusername myusername 4.0K May 10 14:03 ..
drwxr-xr-x  2 myusername           42421 4.0K May  8 15:25 .tmp

Tried to replicate the error I was getting earlier this week and now I'm getting:
==============
STEP 0

18:01:34 - PLAN
run hello world in the ipython notebook
18:01:37 - ACTION
**IPythonRunCellAction**
THOUGHT:Sure, let's run a simple "Hello World!" in the IPython notebook environment:
CODE:
print("Hello World!")
18:01:37 - OBSERVATION
IPythonRunCellObservation(content='bash: line 1: /workspace/.tmp/.ipython_execution_tmp.py: No such file or directory', code='print("Hello World!")', observation='run_ipython')

==============
STEP 1

18:01:40 - ACTION
**IPythonRunCellAction**
THOUGHT:It seems there was an issue with the execution environment. Let me try running the "Hello World!" command again for you.
CODE:
print("Hello World!")
18:01:40 - OBSERVATION
IPythonRunCellObservation(content='bash: line 1: /workspace/.tmp/.ipython_execution_tmp.py: No such file or directory', code='print("Hello World!")', observation='run_ipython')

==============
STEP 2

18:01:46 - ACTION
It appears there is a persistent issue with the IPython notebook execution environment. Unfortunately, I can't directly resolve this environment setup issue from here. However, you can try running the Python code locally on your machine or in any online Python compiler. Here's the code you would use:

```python
print("Hello World!")

You can simply copy this into your Python environment and run it to see the output. If you have any other questions or need further assistance with a different task, feel free to ask!
18:01:46 - opendevin:INFO: agent_controller.py:201 - Setting agent state from AgentState.RUNNING to AgentState.AWAITING_USER_INPUT

When I go to docker desktop, click the vertical ... and "open in terminal" then poked around with these commands and results:

pwd
/workspace
ls
open_devin_test
cd ..
ls
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opendevin  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  workspace
cd opt
ls
pwd
/opt

In other words, when looking inside the docker container for the sandbox, I don't see any subfolders inside `/opt`, so I'm not sure how it's supposed to handle the volume mapping requested with this -v flag: `-v $WORKSPACE_BASE:/opt/workspace_base`

@rbren Shouldn't open_devin_test folder be mapped to /workspace in sandbox?
@williambarberjr  What happens if you set export WORKSPACE_BASE=$(pwd)/workspace;?

@williambarberjr Could you check the main version?

@williambarberjr What happens if you set export WORKSPACE_BASE=$(pwd)/workspace;?

If I understand what the -v volume flag is doing in the docker run command, I believe doing that would just try to map a) the result of concatenating my current working directory with /workspace (a directory which doesn't exist on my local machine) to b) /opt/workspace_base within the docker container (which also doesn't exist within the docker container)
So I don't think that will solve anything unless I'm missing something...

How did  open_devin_test folder come inside the workspace directory?
is that a nested folder?

a directory which doesn't exist on my local machine

The following line will automatically create it.
https://github.com/OpenDevin/OpenDevin/blob/8bfae8413e74e1b6095d5380a7ea93b6b73a4537/containers/app/Dockerfile#L42
Duplicate of #1366

I'm going to dedupe this with https://github.com/OpenDevin/OpenDevin/issues/1366
This is weird stuff, it seems to only pop up for a few folks. I will put more info in #1366 if anyone is able to help debug further


Mismatch In Joint Tessellation Factor

First, in the part of the initialization that estimates the number of coordinates and indices, there is a mistake in estimating the number of joint segments when the join type is round, and it uses the tesselation factor of endpoints for joints. Whereas, during calculate joints coordinates and others it uses the right one. So there is a mismatch.
Second, the tesselation factor is too low. The joints with round join type are the same as the bevel ones. I think there is a need to increase the tesselation factor. But I did not change it. Consider, please.

Thanks! I will tweak the tesselation of joints a bit to make it look smoother.

Thanks! I will tweak the tesselation of joints a bit to make it look smoother.

Thanks too.
Also, I had a problem with lines with a complex pattern like a circle or bordered line.
Those patterns result in something like below:

I couldn't fix this by changing the join tesselation method due to having some problems yet.
Lines gonna be awesome if this issue is fixed.
Thanks for your attention.

These look strange. Could you post a github issue about this? If you could create a code snippet with style and line coordinates, it would be helpful. Something that I could copy-paste into a working sample and see the result.

These look strange. Could you post a github issue about this? If you could create a code snippet with style and line coordinates, it would be helpful. Something that I could copy-paste into a working sample and see the result.

Given the line joint tessellation method that I read, it's expected to see such behavior, as I explain below:

I think the workaround that I didn't implement yet is more flexible than the current way.

Also, I will create the snippet code with resources that shows the problem as soon as I can.


Why Producer not lookup lookupd for produce connection like Consumer ?

Why the implementation of producer not apply  consumer model, get nsqd server address by lookup lookupd servers, then establish a connection for message sending, and decide which nsqd to send a message by scheduling lookup lookupd servers
like this:
`
public NSQConsumer(final NSQLookup lookup, final String topic, final String channel, final NSQMessageCallback callback) {
this(lookup, topic, channel, callback, new NSQConfig());
}
public NSQConsumer(final NSQLookup lookup, final String topic, final String channel, final NSQMessageCallback callback,
                   final NSQConfig config) {
    this(lookup, topic, channel, callback, config, null);
}

public NSQConsumer(final NSQLookup lookup, final String topic, final String channel, final NSQMessageCallback callback,
                   final NSQConfig config, final NSQErrorCallback errCallback) {
    this.lookup = lookup;
    this.topic = topic;
    this.channel = channel;
    this.config = config;
    this.callback = callback;
    this.errorCallback = errCallback;

}

`
but Producer has initialized hardly like this:
`
public class NSQProducer {
private Set addresses = Sets.newConcurrentHashSet();
...
protected Connection getConnection() throws NoConnectionsException {
int c = 0;
while (c < connectionRetries) {
ServerAddress[] serverAddresses = addresses.toArray(new ServerAddress[addresses.size()]);
if (serverAddresses.length != 0) {
try {
return pool.borrowObject(serverAddresses[roundRobinCount++ % serverAddresses.length]);
} catch (NoSuchElementException e) {
try {
Thread.sleep(1000);
} catch (InterruptedException ix) {
throw new NoConnectionsException("Could not acquire a connection to a server", ix);
}
} catch (Exception ex) {
throw new NoConnectionsException("Could not acquire a connection to a server", ex);
}
}
}
throw new IllegalStateException("No server configured for producer");
}
`
I think it's good for Increase/Decrease number of nsqd servers.

There is a huge difference between Consumers and Producers. With Producers you want to control which nodes are used for a specific topic but as a Consumer you really don't have a choice because you want to consume the messages from all nodes of a specific topic. If you think that Producers should use lookupd then you can easily bolt it on yourself.


Running model on webcam

Is it possible to run the model on a webcam feed? Is there some code for this? After reading the README it doesn't seem straight forward.

There is no code for that, no. You would need to manually obtain frames from the webcam feed and input them to the network. I would guess however that TrackR-CNN is simply too slow to be useful for live streams.

Ok. What is the slowest step of TrackR-CNN? Is it having to run the ResNet model three times for t-1, t, and t+1?

Hi,
yes, I think running the backbone should be the slowest step. Additionally the Mask R-CNN second stage is probably also relatively expensive, but I expect it to be still less than the backbone. If you want to know the details you'll have to run a profiler.

I agree. Of course, using a smaller ResNet for example will always increase speed and decrease quality. Closing this for now, please reopen if needed.


Fixes #3260, handle null latlng when adding current location marker

Description (required)
Fixes  NPE in Nearby #3260
What changes did you make and why?
Handled null location when adding current location marker
Tests performed (required)
Tested betaDebug on Google Pizel 2, Api 27

Codecov Report

Merging #3261 into master will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master   #3261      +/-   ##
=========================================
- Coverage    6.54%   6.54%   -0.01%
=========================================
  Files         270     270
  Lines       11613   11619       +6
  Branches      906     906
=========================================
  Hits          760     760
- Misses      10793   10799       +6
  Partials       60      60

Impacted Files
Coverage Δ

...rw/commons/nearby/fragments/NearbyMapFragment.java
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b9e792...69d388c. Read the comment docs.


fix parsing of clock face numbers from path for polymer version 1.2.0

Polymer version  1.2 changed their path handling and are now prepending a '#'
to the number part of the path. This is the Polymer change that broke it:
https://github.com/Polymer/polymer/commit/10021cc51c3f2bb113039d80e6ad1fd35e3f4ed

Hey, sorry, I noticed the issue my self before seeing your PR. Fixed it in d8ec0fb.


Remove copy of date, fix libcurl directly.

Codecov Report

Merging #212 (c1a5cfb) into stable/v8.x (537e8de) will increase coverage by 0.29%.
The diff coverage is n/a.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@               Coverage Diff               @@
##           stable/v8.x     #212      +/-   ##
===============================================
+ Coverage        63.82%   64.12%   +0.29%
===============================================
  Files               72       72
  Lines             4274     4273       -1
  Branches           527      527
===============================================
+ Hits              2728     2740      +12
+ Misses            1317     1303      -14
- Partials           229      230       +1

Components
Coverage Δ

AuthManager
88.38% <ø> (ø)

Cache
28.24% <ø> (+2.25%)
:arrow_up:

FDS
68.32% <ø> (+0.06%)
:arrow_up:

FileWatcher
56.25% <ø> (ø)

Flip
67.42% <ø> (ø)

gRPC
77.12% <ø> (ø)

Logging
28.66% <ø> (ø)

Metrics
80.20% <ø> (ø)

Options
100.00% <ø> (ø)

Setting
56.68% <ø> (ø)

StatusObject
73.83% <ø> (ø)

Utility
83.44% <ø> (ø)

Version
95.83% <ø> (ø)


adding tutorial

After review and discussion, this would make more sense as a How-To. Do you mind closing this request and adding a pull request to the How-To repo? There are no changes required.
Thanks!
Meredith

Moved to how-to tutorials
https://github.com/SAPDocuments/How-Tos/pull/241


meeting

Did we schedule a meeting for this week to discuss your project as a group?

Hi Jarrod! I'm so sorry. I forgot to create the issue here GitHub to track the meeting.
The meeting is 9:00 to 10:00 am this coming Monday.

Hi @jarrodmillman,
Thank you for your time!
I didn't add you in my reply above so I added you here to make sure you can receive my reply.
You mentioned that you want to me to set a meeting time on GitHub. Is opening this issue sufficient (I can't find a specific calendar feature on GitHub)?
About the meeting, we plan to present to you the very clear flow of our entire project (on whiteboard or otherwise). Then we will show you some results and considerations for each component along our analysis. We don't plan to get down to the specifics of code implementation in the meeting but will try to push out as much as work onto GitHub before tonight.
Enjoy your weekend!
Team Gamma


made libc optional so that it isnt pulled when not using libstd

Currently when compiling wasmi to targets that do not support libc, building fails even when using the core feature instead of the std feature. This PR fixes that :smiley:

[clabot:check]

Reviewers, please note that the CI failed because the rustfmt component is missing for the nightly compiler configured in your settings.

Tagging recent participants in the git tree:
@bddap @pepyakin @taegyunkim @mystor

wasm32-unknown-unknown, for one :smile:.
Specifically I'm trying to compile wasmi as no_std into an SGX enclave using a custom target provided by the Baidu Rust SGX SDK... but you can test this by compiling to any target that does not support libc, which is about half the tier-2 targets.

Hm, both
cargo +nightly build --target x86_64-fortanix-unknown-sgx --no-default-features --features core
cargo build --target wasm32-unknown-unknown --no-default-features --features core

work for me without this PR.

interesting... perhaps libc has some support for them in case it is being compiled to those targets..?
In any case, this PR should improve compile times for users of no_std because libc won't be pulled and built when wasmi isn't using it

Yup, this is indeed the case:
https://github.com/rust-lang/libc/blob/7477c68275b1b6da3e8ce81d0545c9484236b35f/src/lib.rs#L145
libc provides support for compiling to official targets.